Skip to content

Your community in search engines and AI

The robots.txt, sitemap.xml and llms.txt every community serves on its own, what Google and AI assistants read and what stays out, how each space's visibility changes that, the search description, and how to submit the sitemap in Google Search Console and Bing Webmaster Tools.

Every Memberfy community, on its Memberfy address or on its custom domain, already gives search engines and AI assistants the files they look for. You don't need to create or update anything: the files are built from the community's menu as it looks to someone who is not signed in.

FileWhat it has
/robots.txtGoogle, Bing and other crawlers. What they shouldn't visit: only the API, the checkout, administration and the account. Everything else is allowed
/sitemap.xmlSearch engines. The list of pages to index: the home page, the open spaces, the upcoming public events and the plans page (/pricing)
/llms.txtAI assistants (ChatGPT, Claude, Perplexity…). The community in Markdown: what is open, with a link; what is closed, only with its name and what it takes to get in
/llms-full.txtAI assistants. The same, with full descriptions instead of one line

Example: https://community.papodesysadmin.org/sitemap.xml or https://papodesysadmin.memberfy.net/llms.txt.

What it's for

  • Getting found: people searching Google for your community's topic land on the open spaces, the events and the plans.
  • Being described well by AI: when someone asks an assistant about your community, it has an organized source, with what is public, the prices and the way in.
  • Without exposing what is closed: the content of a restricted area never goes in; at most its name and what it takes to see it.

How it works

What is indexed and what isn't

In the sitemap and the indexNot in it
The home pageAdministration, settings, checkout and the account (blocked in robots.txt)
The spaces open to people who aren't signed in (Public visibility)Padlocked spaces: they show up in llms.txt by name only, and the page asks search engines not to index it
The upcoming public events in those spacesHidden spaces and group spaces
The plans page, when a plan is for saleLink only plans
Invite, unsubscribe and payment links, forms opened only by link, and member pages

Link spaces aren't in the sitemap (the page belongs to another site), but they show up in llms.txt with their target address.

A page that doesn't exist answers 404. An address with a section or space that doesn't exist (or that someone not signed in can't see) returns not found to search engines instead of a copy of the home page. That way an old link drops out of the index on its own.

How each space's visibility affects this

What counts is the space's and the section's visibility, and the For those without access option:

The space, for someone not signed inSitemapllms.txt
PublicInUnder Public, with link and description
Members, with a padlockOutUnder Requires sign-in or purchase: the name, the description and "requires an account"
Subscribers, or Private with a plan, with a padlockOutThe name and "requires a plan", linking to the plans page
Private with a product, with a padlockOutThe name and "requires a purchase"
Private for groups onlyOutNot listed: there's no way in from outside
Hide under For those without accessOutNot listed

If the community has no space open to people who aren't signed in, llms.txt has only the name, the description and "private community; the content requires an account".

In llms.txt, open Events spaces list the upcoming events with date and time; Store spaces, the products with prices; and the Plans section has each public plan with the prices of its billing options. The file is in the community's language (Settings › General › Language).

Want a space to show up on Google? Make it Public. Want it not to show up even by name? Use Hide under For those without access.

The search description

The description Google shows under the title, which also opens llms.txt, comes, in this order:

  1. From the SEO Description, in Settings › Search and discovery › SEO Meta Tags (see Images and SEO);
  2. If that's empty, from the Community Description, in Settings › General;
  3. If both are empty, from a default sentence: "Papo de Sysadmin: a community on Memberfy."

The default sentence says nothing about the topic, and the description is what search engines and AI use to decide who to show the community to. Write one or two sentences saying what the community is, who it's for and what people find in it.

When it changes

The files follow the community: a new open space, a new event or a new plan shows up within 5 minutes. Search engines reread on their own schedule, from a few hours to a few days.

Who can

The files are automatic and have no settings. The owner and the admin change what appears in them, through the spaces' visibility, the plans and the description.

Step by step: Google Search Console

Search Console shows which pages Google indexed, the searches that bring people in, and errors. Verification needs DNS access, so it applies to a custom domain; on the Memberfy address, Google finds the sitemap on its own through robots.txt.

Role: whoever manages the domain's DNS.

  1. Open search.google.com/search-console and click Add property.
  2. Choose Domain and type the main domain, without https:// and without the subdomain: papodesysadmin.org.
  3. Google shows a TXT record. Create it in your DNS provider's panel (Cloudflare, GoDaddy, Namecheap…), under the name @ (or the root domain), and click Verify. DNS can take a few minutes.
  4. In the left menu, open Sitemaps, type the full address, https://community.papodesysadmin.org/sitemap.xml, and click Submit.
  5. Come back in a few days: Pages shows what was indexed and why each page was left out.

Step by step: Bing Webmaster Tools

Bing also feeds other search engines and assistants that use its index.

  1. Open bing.com/webmasters and sign in with a Microsoft, Google or Facebook account.
  2. The shortest path is Import from Google Search Console: it brings the site and the sitemap already verified. To do it by hand, add https://community.papodesysadmin.org and choose verification by DNS record (CNAME).
  3. Under Sitemaps, click Submit sitemap and paste https://community.papodesysadmin.org/sitemap.xml.

Examples

Excel School, on school.excelpractice.com. The Start here section is Public, with the Welcome and Free lessons spaces; Student area is Private with the Complete plan, with a padlock; March class is Private for the class group only. Result:

  • The sitemap has the home page, Welcome, Free lessons, the upcoming events from the open calendar and /pricing.
  • llms.txt lists the two open spaces, the events, the Complete plan with its monthly and yearly price and, under Requires sign-in or purchase, Student area — requires a plan. March class doesn't appear.

Tips

  • Write the Community Description even if you fill in the SEO Description: the first one also shows inside the community.
  • Give every open space a description: it becomes the page's description on Google and the space's line in llms.txt.
  • Use a padlock (not Hide) on the paid areas you want to sell: the name and description reach AI with the way in, without revealing the content.
  • With a custom domain, submit the custom domain's sitemap, not the Memberfy address's: that's the one the pages point to as the official address.

Common errors and how to fix them

SituationWhat to do
A space isn't in the sitemapCheck that it's Public, and that its section is too: a space is never more open than its section
Search shows "a community on Memberfy" as the descriptionFill in the Community Description or the SEO Description and wait for Google to reread the page
Search Console says "Blocked by robots.txt" on a pageIt's an administration, checkout or account page: it shouldn't be indexed
Search Console says "Excluded by 'noindex' tag"It's a padlocked space, a member page or an invite link: it's right to leave it out
Search Console says "Not found (404)"A section or space was deleted, renamed or closed. If you renamed it, the new address goes into the sitemap on its own
I can't verify the .memberfy.net addressVerification needs the DNS, which belongs to Memberfy. Use a custom domain; without one, search engines find the sitemap through robots.txt

FAQ

Can I turn off the sitemap or llms.txt? No. They only have what anyone already sees without joining the community. To take a space out, change its visibility or use Hide.

Does the content of posts and lessons go to AI? Not from closed areas: only the name and what it takes to get in. For Public spaces, the file has the name, the description and the link; an AI that follows the link reads the page like any visitor.

Can I write my own robots.txt? No. Memberfy's blocks only the private areas and allows the rest.

Are the prices in llms.txt the ones on the plans page? Yes, the same, in each option's currency. Link only plans aren't included.