Source Types

BetterPost supports a wide variety of content sources through its modular architecture. The system automatically detects the best way to process each source using specialized processors and fetchers.

Overview

BetterPost uses a sophisticated modular system for content processing:

1. Source Processors - Handle different types of content sources and extract story URLs
2. Story Fetchers - Fetch and parse content from individual URLs

This architecture ensures optimal performance and reliability for each content type. We support these kinds of URLs for content fetching.

Available Source Types

BetterPost offers a simplified set of source types that cover most use cases:

• Link - Any URL with automatic feed discovery and web processing
• Web Search - General web search results
• YouTube Search - YouTube video content
• X/Twitter Search - Twitter/X search results
• Bluesky Search - Bluesky search results

Link (Auto-Detect)

The most versatile source type that automatically discovers and processes any URL.

Input Format: Any website URL

https://example.com/blog
https://company.com/articles
https://publication.com
https://twitter.com/username
https://www.reddit.com/r/subreddit
https://news.google.com/search?q=topic

How it works:

1. RSS/Atom Discovery - Automatically finds RSS/Atom feeds in the page's HTML
2. Social Media Detection - Recognizes Twitter/X profiles, Reddit subreddits, and other social platforms
3. Web Processing - Falls back to intelligent web processing for blog content
4. Search Processing - Handles Google News and other search result pages
5. Video Content - Processes YouTube and other video platform URLs

Advantages:

• Works with any website or platform
• Automatically discovers the best processing method
• No need to know the specific source type
• Intelligent fallback mechanisms
• Handles platform changes automatically

Best For:

• Any website or blog
• Social media accounts and searches
• News sites and publications
• Video platforms
• When you're unsure about the specific source type

Fetch Frequency: Daily (recommended)

Web Search

Monitors general web search results for specific terms.

Input Format: Search query terms (not a URL)

"nextjs tutorial" 2024
"react best practices" -2023
site:dev.to typescript
machine learning news

Advantages:

• Discovers content from across the web
• Finds new and emerging sources
• Captures blog posts and articles
• Comprehensive topic coverage

Best For:

• Comprehensive topic monitoring
• Finding new content sources
• Research and trend monitoring
• Industry news discovery

Fetch Frequency: Daily

YouTube Search

Monitors YouTube for new videos matching search terms.

Input Format: YouTube search terms

nextjs tutorial
"react hooks" 2024
javascript tips

Advantages:

• Video content monitoring
• Educational and tutorial content
• Visual and demonstration content
• Quality filtering (25,000+ views, 4+ minutes)

Best For:

• Educational content
• Product demos and tutorials
• Conference talks and presentations
• Technical video content

Fetch Frequency: Daily

X/Twitter Search

Monitors Twitter/X search results for specific keywords or hashtags.

Input Format: Twitter/X search terms

#nextjs
"machine learning" -ads

Advantages:

• Catches content from many users
• Good for trending topics
• Can filter by various criteria
• High engagement filtering (only posts with 1000+ likes/retweets)

Best For:

• Industry hashtags and trends
• Event coverage
• Community discussions
• Brand monitoring

Fetch Frequency: Hourly (recommended for time-sensitive content)

Bluesky Search

Monitors Bluesky search results for specific keywords or hashtags.

Input Format: Bluesky search terms

#nextjs
"machine learning" -ads

Advantages:

• Catches content from many users
• Good for trending topics
• Can filter by various criteria
• High engagement filtering (only posts with 500+ likes)

Best For:

• Industry hashtags and trends
• Event coverage
• Community discussions
• Brand monitoring

Fetch Frequency: Hourly (recommended for time-sensitive content)

Configuration Options

Fetch Frequency

Different source types have recommended fetch frequencies:

• Hourly: Social media searches (Twitter/X)
• Daily: Most other sources (web search, YouTube, general URLs)
• Weekly: Low-volume or archival sources

Source Processing

BetterPost automatically selects the appropriate processor and fetcher:

1. URL Analysis - Determines the best processor for the input
2. Content Discovery - Extracts story URLs and metadata
3. Content Fetching - Uses specialized fetchers for each URL type
4. Content Processing - Parses and summarizes the content

Choosing the Right Source Type

For News and Blogs

1. First choice: Link (auto-detect) - handles RSS feeds automatically
2. Discovery: Web Search (to find new sources)

For Social Media

1. Specific accounts: Link (auto-detect) - recognizes social media URLs
2. Topic monitoring: X/Twitter Search, Web Search

For Technical Content

1. RSS feeds from developer blogs (use Link)
2. Reddit subreddits (use Link)
3. Twitter accounts of tech leaders (use Link)
4. YouTube searches for tutorials and talks

For Industry Research

1. RSS feeds from industry publications (use Link)
2. Twitter searches for industry hashtags
3. Web searches for recent reports and studies
4. Google News (use Link)

For Video Content

1. YouTube searches for educational content
2. Video URLs for specific content monitoring (use Link)

Source Health and Monitoring

Health Indicators

Each source type has different failure patterns:

• RSS Feeds: Usually very reliable, may fail if site changes feed URL
• Social Media: May break due to platform changes or rate limiting
• Web Processing: Most fragile, sensitive to site changes
• Search Results: Generally stable but may return varying quality
• Video Content: Reliable but may have access restrictions

Troubleshooting by Source Type

Link (Auto-Detect)

• No content: Check if URL is still valid
• Old content: Source may have moved or been discontinued
• Processing errors: May need manual intervention

Web Search

• Irrelevant results: Refine search terms
• Too much content: Add exclusion terms
• No results: Broaden search terms

YouTube Search

• Access restrictions: Some videos may be private or region-locked
• Processing errors: Video format or platform changes
• Rate limiting: May need to reduce fetch frequency

X/Twitter Search

• Rate limiting: Reduce fetch frequency
• Authentication: May need different approach
• Platform changes: May need manual intervention

Advanced Tips

RSS Feed Discovery

Many sites have hidden RSS feeds that Link will automatically discover:

https://example.com/feed
https://example.com/rss
https://example.com/atom.xml
https://example.com/feeds/all.atom.xml

Social Media Best Practices

• Monitor key accounts with Link (auto-detect)
• Use searches for broader topic coverage
• Combine multiple platforms for comprehensive coverage
• Use engagement filters to focus on quality content

Web Processing Optimization

• Use Link processor for automatic feed discovery
• Monitor for site changes that break processing
• Consider reaching out to sites to request RSS feeds

Search Query Optimization

• Use quotes for exact phrases: "machine learning"
• Exclude unwanted terms: AI -chatbot
• Use site-specific searches: site:dev.to react
• Include time constraints: 2024 OR 2023

Video Content Optimization

• Use specific search terms for better results
• Consider video length and quality filters
• Monitor for new video platforms and formats

Performance Considerations

Resource Usage by Source Type

Most Efficient:

1. RSS/Atom feeds (discovered by Link)
2. Social media APIs
3. Simple web pages

Moderate:

1. Video content processing
2. Search result processing
3. Feed discovery and validation

Least Efficient:

1. Complex web processing
2. JavaScript-heavy sites
3. Video content with processing

Optimization Tips

• Use Link (auto-detect) when possible - it automatically finds the most efficient method
• Use appropriate fetch frequencies
• Monitor source performance regularly
• Remove consistently failing sources
• Group similar sources for batch processing
• Use engagement filters to reduce noise