Adding Sources

Content sources are the foundation of BetterPost. They define where you monitor content from - RSS feeds, social media accounts, blogs, and more. This guide covers everything you need to know about adding and managing sources.

What are Sources?

A source is any location where BetterPost monitors for new content. Sources can be:

• RSS/Atom Feeds - Traditional syndication feeds from blogs and news sites
• Social Media - Twitter/X accounts, Reddit subreddits, Bluesky feeds
• Websites - Blog pages with automatic feed discovery
• Search Results - Monitor search results from various platforms
• Video Platforms - YouTube and other video content
• News & Finance - Google News and Google Finance

Each source is processed through specialized processors and fetchers to ensure optimal content discovery and retrieval.

Two Ways to Add Sources

1. AI-Powered Source Search

The fastest way to find relevant sources is using our AI-powered search:

1. Click "Search for Sources" in your Sources page
2. Choose a search method:
   • AI-Generated Terms - We analyze your project and suggest relevant search terms
   • Manual Search - Enter your own search terms
3. Review Results - See potential sources with relevance scores
4. Add Sources - Click the button to add sources directly

The AI search automatically:

• Detects the best source type (RSS feed, social media, etc.)
• Finds RSS feeds when available
• Configures optimal fetch frequencies
• Sets up source names and descriptions

2. Manual Source Addition

For more control, you can add sources manually:

1. Click "Add Source" in your Sources page
2. Choose source type:
   • Link - Any URL (auto-detects the best processing method)
   • Web Search - Search query terms
   • YouTube Search - YouTube search terms
   • X/Twitter Search - Twitter/X search terms
   • Bluesky Search - Bluesky search terms
3. Enter source details:
   • URL or search terms (depends on source type)
   • Fetch frequency (how often to check for updates)
4. Save - BetterPost will validate and activate the source

3. Bulk Import

For adding multiple sources at once:

1. Click "Bulk Import" in your Sources page
2. Paste your list of URLs or identifiers (one per line)
3. Review the detected types - BetterPost will show what each source will be processed as
4. Confirm import - All sources will be added with appropriate settings

Source Processing Architecture

BetterPost uses a sophisticated two-tier system for processing sources:

Source Processors

Specialized processors handle different types of content sources, automatically detecting the best approach for each source type.

Story Fetchers

Specialized fetchers retrieve and parse content from individual URLs. We support various URL types for content fetching, ensuring reliable content retrieval regardless of the source format.

Source Detection

When you add a source, BetterPost automatically analyzes the input to determine the best processing approach:

Automatic Detection Process

1. Input Analysis - Determines the appropriate processor for the input
2. Content Discovery - Extracts story URLs and metadata from the source
3. Content Fetching - Uses specialized fetchers to retrieve and parse content
4. Quality Filtering - Applies engagement filters for social media content

What Gets Detected

• Feed URLs - Discovers hidden RSS/Atom feeds from websites
• Source Type - Determines the optimal monitoring method
• Content Title - Extracts site name or publication title
• Update Frequency - Suggests optimal checking intervals
• Quality Filters - Applies engagement thresholds for social content

Source Configuration Options

Fetch Frequency

Controls how often BetterPost checks for new content:

• Hourly - Best for high-volume sources like social media
• Daily - Good for most blogs and news sites
• Weekly - Suitable for infrequently updated sources

Quality Filtering

Social media sources automatically filter content by engagement:

• Twitter/X Search - Only posts with 1000+ likes or retweets
• Reddit Subreddits - Only posts with 500+ upvotes
• Bluesky Search - Only posts with 500+ likes
• YouTube Search - Only videos with 25,000+ views and 4+ minutes
• Instagram Users - Content filtered by engagement metrics
• LinkedIn - Professional content with engagement filtering

Source Management

Source Status Indicators

Each source shows its current status:

• Active - Working normally
• Warning - Minor issues, still functional
• Error - Failed to fetch content

Manual Refresh

You can manually refresh any source:

1. Find the source in your Sources list
2. Click the actions menu (three dots)
3. Select "Refresh Now" - Forces an immediate content check

Editing Sources

To modify a source:

1. Click the source name or use the actions menu
2. Edit any field:
   • Source name
   • URL/identifier
   • Fetch frequency
3. Save changes - Updates take effect immediately

Deleting Sources

To remove a source:

1. Click the actions menu (three dots)
2. Select "Delete"
3. Confirm deletion - This cannot be undone

Note: Deleting a source doesn't remove stories that were already discovered from that source.

Best Practices

Quality Over Quantity

• Start with 5-10 high-quality sources
• Add more gradually as you understand what works
• Remove sources that consistently produce irrelevant content

Source Diversity

Mix different types of sources:

• Primary sources - Original publications in your field
• Aggregators - Curated collections and industry roundups
• Social media - Real-time discussions and trending topics
• Search feeds - Catch content from smaller or newer sources

Monitor Source Health

• Check source status regularly
• Remove or fix broken sources quickly
• Adjust fetch frequencies based on source update patterns

Optimize for Your Workflow

• Use hourly updates for time-sensitive content
• Set daily updates for most sources to reduce noise
• Use weekly updates for research or historical content

Common Source Types

RSS/Atom Feeds

• Example: https://example.com/feed.xml
• Best for: Blogs, news sites, podcasts
• Pros: Reliable, structured, efficient
• Cons: Not all sites provide feeds

Any URL (Feed Discovery)

• Example: https://example.com/blog
• Best for: Modern blogs without obvious RSS feeds
• Pros: Automatic feed discovery, works with any site
• Cons: Slightly slower than direct RSS feeds

Twitter/X Accounts

• Example: https://twitter.com/username or username
• Best for: Industry leaders, company accounts, news sources
• Pros: Real-time updates, high engagement
• Cons: Limited by platform changes

Twitter/X Search

• Example: https://twitter.com/search?q=%23nextjs or #nextjs
• Best for: Trending topics, industry hashtags
• Pros: Catches content from many users, engagement filtering
• Cons: Can be high volume

Bluesky Search

• Example: https://bsky.app/search?q=%23nextjs or #nextjs
• Best for: Decentralized social media trends, community discussions
• Pros: High-quality community moderation, engagement filtering
• Cons: Smaller user base compared to Twitter

Reddit Subreddits

• Example: https://reddit.com/r/technology or technology
• Best for: Community discussions, trending topics
• Pros: High-quality community moderation, engagement filtering
• Cons: Can be very high volume

YouTube Search

• Example: https://www.youtube.com/results?search_query=nextjs or nextjs tutorial
• Best for: Educational content, tutorials, conference talks
• Pros: Video content, quality filtering
• Cons: Limited to video content

Web Search

• Example: "machine learning news" or site:dev.to react
• Best for: Comprehensive topic monitoring, finding new sources
• Pros: Discovers content from across the web
• Cons: Results can vary in quality

Google News

• Example: https://news.google.com/search?q=artificial+intelligence
• Best for: Industry news monitoring, breaking news
• Pros: Curated news content, multiple source aggregation
• Cons: Limited to news content

Google Finance

• Example: https://www.google.com/finance/quote/AAPL
• Best for: Company monitoring, market research
• Pros: Company-specific news, financial data
• Cons: Limited to financial content

Google News

• Example: https://news.google.com/search?q=artificial+intelligence
• Best for: Industry news monitoring, breaking news
• Pros: Curated news content, multiple source aggregation
• Cons: Limited to news content

Instagram Users

• Example: https://instagram.com/username or username
• Best for: Visual content, brand monitoring, influencer tracking
• Pros: Rich visual content, high engagement
• Cons: Limited text content, platform restrictions

LinkedIn

• Example: https://linkedin.com/company/company-name or company-name
• Best for: Professional content, B2B insights, industry updates
• Pros: Professional focus, business-relevant content
• Cons: Limited public content access

Troubleshooting

Source Won't Add

• Check the URL - Make sure it's accessible and correct
• Try different format - Some sources work better with different input formats
• Check for typos - URLs and usernames must be exact

No Content Being Found

• Check source health - Look for error messages
• Verify the source - Visit the URL manually to confirm it has content
• Review criteria - Content might be filtered out by your project criteria
• Check quality filters - Social media content may be filtered by engagement

Too Much Irrelevant Content

• Refine project criteria - Make them more specific
• Remove noisy sources - Some sources may not be a good fit
• Adjust source selection - Focus on higher-quality sources
• Use quality filters - Social media sources automatically filter by engagement

Source Keeps Breaking

• Check for URL changes - Sites sometimes change their feed URLs
• Verify platform changes - Social media platforms may change their APIs
• Contact support - Some sources may require special handling

Advanced Tips

Finding Hidden Feeds

Many sites have RSS feeds that aren't prominently displayed:

• Try adding /feed, /rss, or /atom.xml to the URL
• Check the page source for <link rel="alternate">
• Use BetterPost's automatic discovery with Any URL processor

Optimizing Performance

• Group similar sources with the same fetch frequency
• Use appropriate source types for your content needs
• Monitor source performance and adjust as needed

Content Filtering

• Use specific project criteria to filter at the source level
• Consider separate projects for different content types
• Regularly review and refine your content criteria
• Leverage automatic quality filtering for social media sources