Your community in search engines and AI
The robots.txt, sitemap.xml and llms.txt every community serves on its own, what Google and AI assistants read and what stays out, how each space's visibility changes that, the search description, and how to submit the sitemap in Google Search Console and Bing Webmaster Tools.
Every Memberfy community, on its Memberfy address or on its custom domain, already gives search engines and AI assistants the files they look for. You don't need to create or update anything: the files are built from the community's menu as it looks to someone who is not signed in.
| File | What it has |
|---|---|
| /robots.txt | Google, Bing and other crawlers. What they shouldn't visit: only the API, the checkout, administration and the account. Everything else is allowed |
| /sitemap.xml | Search engines. The list of pages to index: the home page, the open spaces, the upcoming public events and the plans page (/pricing) |
| /llms.txt | AI assistants (ChatGPT, Claude, Perplexity…). The community in Markdown: what is open, with a link; what is closed, only with its name and what it takes to get in |
| /llms-full.txt | AI assistants. The same, with full descriptions instead of one line |
Example: https://community.papodesysadmin.org/sitemap.xml or https://papodesysadmin.memberfy.net/llms.txt.
What it's for
- Getting found: people searching Google for your community's topic land on the open spaces, the events and the plans.
- Being described well by AI: when someone asks an assistant about your community, it has an organized source, with what is public, the prices and the way in.
- Without exposing what is closed: the content of a restricted area never goes in; at most its name and what it takes to see it.
How it works
What is indexed and what isn't
| In the sitemap and the index | Not in it |
|---|---|
| The home page | Administration, settings, checkout and the account (blocked in robots.txt) |
| The spaces open to people who aren't signed in (Public visibility) | Padlocked spaces: they show up in llms.txt by name only, and the page asks search engines not to index it |
| The upcoming public events in those spaces | Hidden spaces and group spaces |
| The plans page, when a plan is for sale | Link only plans |
| Invite, unsubscribe and payment links, forms opened only by link, and member pages |
Link spaces aren't in the sitemap (the page belongs to another site), but they show up in llms.txt with their target address.
A page that doesn't exist answers 404. An address with a section or space that doesn't exist (or that someone not signed in can't see) returns not found to search engines instead of a copy of the home page. That way an old link drops out of the index on its own.
How each space's visibility affects this
What counts is the space's and the section's visibility, and the For those without access option:
| The space, for someone not signed in | Sitemap | llms.txt |
|---|---|---|
| Public | In | Under Public, with link and description |
| Members, with a padlock | Out | Under Requires sign-in or purchase: the name, the description and "requires an account" |
| Subscribers, or Private with a plan, with a padlock | Out | The name and "requires a plan", linking to the plans page |
| Private with a product, with a padlock | Out | The name and "requires a purchase" |
| Private for groups only | Out | Not listed: there's no way in from outside |
| Hide under For those without access | Out | Not listed |
If the community has no space open to people who aren't signed in, llms.txt has only the name, the description and "private community; the content requires an account".
In llms.txt, open Events spaces list the upcoming events with date and time; Store spaces, the products with prices; and the Plans section has each public plan with the prices of its billing options. The file is in the community's language (Settings › General › Language).
Want a space to show up on Google? Make it Public. Want it not to show up even by name? Use Hide under For those without access.
The search description
The description Google shows under the title, which also opens llms.txt, comes, in this order:
- From the SEO Description, in Settings › Search and discovery › SEO Meta Tags (see Images and SEO);
- If that's empty, from the Community Description, in Settings › General;
- If both are empty, from a default sentence: "Papo de Sysadmin: a community on Memberfy."
The default sentence says nothing about the topic, and the description is what search engines and AI use to decide who to show the community to. Write one or two sentences saying what the community is, who it's for and what people find in it.
When it changes
The files follow the community: a new open space, a new event or a new plan shows up within 5 minutes. Search engines reread on their own schedule, from a few hours to a few days.
Who can
The files are automatic and have no settings. The owner and the admin change what appears in them, through the spaces' visibility, the plans and the description.
Step by step: Google Search Console
Search Console shows which pages Google indexed, the searches that bring people in, and errors. Verification needs DNS access, so it applies to a custom domain; on the Memberfy address, Google finds the sitemap on its own through robots.txt.
Role: whoever manages the domain's DNS.
- Open search.google.com/search-console and click Add property.
- Choose Domain and type the main domain, without
https://and without the subdomain:papodesysadmin.org. - Google shows a TXT record. Create it in your DNS provider's panel (Cloudflare, GoDaddy, Namecheap…), under the name
@(or the root domain), and click Verify. DNS can take a few minutes. - In the left menu, open Sitemaps, type the full address,
https://community.papodesysadmin.org/sitemap.xml, and click Submit. - Come back in a few days: Pages shows what was indexed and why each page was left out.
Step by step: Bing Webmaster Tools
Bing also feeds other search engines and assistants that use its index.
- Open bing.com/webmasters and sign in with a Microsoft, Google or Facebook account.
- The shortest path is Import from Google Search Console: it brings the site and the sitemap already verified. To do it by hand, add
https://community.papodesysadmin.organd choose verification by DNS record (CNAME). - Under Sitemaps, click Submit sitemap and paste
https://community.papodesysadmin.org/sitemap.xml.
Examples
Excel School, on school.excelpractice.com. The Start here section is Public, with the Welcome and Free lessons spaces; Student area is Private with the Complete plan, with a padlock; March class is Private for the class group only. Result:
- The sitemap has the home page, Welcome, Free lessons, the upcoming events from the open calendar and
/pricing. llms.txtlists the two open spaces, the events, the Complete plan with its monthly and yearly price and, under Requires sign-in or purchase, Student area — requires a plan. March class doesn't appear.
Tips
- Write the Community Description even if you fill in the SEO Description: the first one also shows inside the community.
- Give every open space a description: it becomes the page's description on Google and the space's line in
llms.txt. - Use a padlock (not Hide) on the paid areas you want to sell: the name and description reach AI with the way in, without revealing the content.
- With a custom domain, submit the custom domain's sitemap, not the Memberfy address's: that's the one the pages point to as the official address.
Common errors and how to fix them
| Situation | What to do |
|---|---|
| A space isn't in the sitemap | Check that it's Public, and that its section is too: a space is never more open than its section |
| Search shows "a community on Memberfy" as the description | Fill in the Community Description or the SEO Description and wait for Google to reread the page |
| Search Console says "Blocked by robots.txt" on a page | It's an administration, checkout or account page: it shouldn't be indexed |
| Search Console says "Excluded by 'noindex' tag" | It's a padlocked space, a member page or an invite link: it's right to leave it out |
| Search Console says "Not found (404)" | A section or space was deleted, renamed or closed. If you renamed it, the new address goes into the sitemap on its own |
I can't verify the .memberfy.net address | Verification needs the DNS, which belongs to Memberfy. Use a custom domain; without one, search engines find the sitemap through robots.txt |
FAQ
Can I turn off the sitemap or llms.txt?
No. They only have what anyone already sees without joining the community. To take a space out, change its visibility or use Hide.
Does the content of posts and lessons go to AI? Not from closed areas: only the name and what it takes to get in. For Public spaces, the file has the name, the description and the link; an AI that follows the link reads the page like any visitor.
Can I write my own robots.txt?
No. Memberfy's blocks only the private areas and allows the rest.
Are the prices in llms.txt the ones on the plans page?
Yes, the same, in each option's currency. Link only plans aren't included.