Aller au contenu

Your community in search engines and AI

The robots.txt, sitemap.xml and llms.txt every community serves on its own, what Google and AI assistants read and what stays out, how each space's visibility changes that, the search description, and how to submit the sitemap in Google Search Console and Bing Webmaster Tools.

Cet article n’a pas encore été traduit. Vous lisez la version en anglais.

Every Memberfy community, on its Memberfy address or on its custom domain, already gives search engines and AI assistants the files they look for. You don't need to create or update anything: the files are built from the community's menu as it looks to someone who is not signed in.

FileWhat it has
/robots.txtGoogle, Bing and other crawlers. What they shouldn't visit: only the API, the checkout, administration and the account. Everything else is allowed
/sitemap.xmlSearch engines. The list of pages to index: the home page, the open spaces, the upcoming public events and the plans page (/pricing) Also the open courses, the calls for papers with submissions open and the certificates their holders let search engines show.
/llms.txtAI assistants (ChatGPT, Claude, Perplexity…). The community in Markdown: what is open, with a link; what is closed, only with its name and what it takes to get in
/llms-full.txtAI assistants. The same, with full descriptions instead of one line

Example: https://community.papodesysadmin.org/sitemap.xml or https://papodesysadmin.memberfy.net/llms.txt.

What it's for

  • Getting found: people searching Google for your community's topic land on the open spaces, the events and the plans.
  • Being described well by AI: when someone asks an assistant about your community, it has an organized source, with what is public, the prices and the way in.
  • Without exposing what is closed: the content of a restricted area never goes in; at most its name and what it takes to see it.

How it works

What is indexed and what isn't

In the sitemap and the indexNot in it
The home pageAdministration, settings, checkout and the account (blocked in robots.txt)
The spaces open to people who aren't signed in (Public visibility)Padlocked spaces: they show up in llms.txt by name only, and the page asks search engines not to index it
The upcoming public events in those spacesHidden spaces and group spaces
The plans page, when a plan is for saleLink only plans
Published courses in open Courses spaces
Calls for papers with submissions open, in open Call for papers spaces
Certificates whose holder turned on Show in search engines (the verification page, without the name or e-mail)
Invite, unsubscribe and payment links, forms opened only by link, and member pages

With Guest Access off ("Allow visitors to view public content", the default), the community publishes nothing in these files beyond the home page, not even indexable certificates: search engines and AI tools only see the home page. See Sign-up and sign-in.

Link spaces aren't in the sitemap (the page belongs to another site), but they show up in llms.txt with their target address.

A page that doesn't exist answers 404. An address with a section or space that doesn't exist (or that someone not signed in can't see) returns not found to search engines instead of a copy of the home page. That way an old link drops out of the index on its own.

How each space's visibility affects this

What counts is the space's and the section's visibility, and the For those without access option:

The space, for someone not signed inSitemapllms.txt
PublicInUnder Public, with link and description
Members, with a padlockOutUnder Requires sign-in or purchase: the name, the description and "requires an account"
Subscribers, or Private with a plan, with a padlockOutThe name and "requires a plan", linking to the plans page
Private with a product, with a padlockOutThe name and "requires a purchase"
Private for groups onlyOutNot listed: there's no way in from outside
Hide under For those without accessOutNot listed

If the community has no space open to people who aren't signed in, llms.txt has only the name, the description and "private community; the content requires an account".

In llms.txt, open Events spaces list the upcoming events with date and time; Store spaces, the products with prices; and the Plans section has each public plan with the prices of its billing options. The file is in the community's language (Settings › General › Language).

Want a space to show up on Google? Make it Public. Want it not to show up even by name? Use Hide under For those without access.

The search description

The description Google shows under the title, which also opens llms.txt, comes, in this order:

  1. From the SEO Description, in Settings › Search and discovery › SEO Meta Tags (see Images and SEO);
  2. If that's empty, from the Community Description, in Settings › General;
  3. If both are empty, from a default sentence: "Papo de Sysadmin: a community on Memberfy."

The default sentence says nothing about the topic, and the description is what search engines and AI use to decide who to show the community to. Write one or two sentences saying what the community is, who it's for and what people find in it.

When it changes

The files follow the community: a new open space, a new event or a new plan shows up within 5 minutes. Search engines reread on their own schedule, from a few hours to a few days.

Who can

The files are automatic and have no settings. The owner and the admin change what appears in them, through the spaces' visibility, the plans and the description.

Verify the community on Google and Bing

On the Memberfy address (.memberfy.net) there is no DNS to change: verification is by HTML tag. It also works on a custom domain.

Role: owner or admin.

  1. In Google Search Console, add the property as a URL prefix (https://papodesysadmin.memberfy.net) and choose the HTML tag method. In Bing Webmaster Tools, choose Meta tag.
  2. Copy the code or the whole tag: <meta name="google-site-verification" content="…"> on Google, <meta name="msvalidate.01" content="…"> on Bing.
  3. In the community, under Administration › Settings › Search and discovery, in the Search engine verification card, paste it in the Google Search Console or Bing Webmaster Tools field and save. Only the code is kept (up to 100 letters, digits, - or _); another tag or other text shows an error on the field. Clearing the field removes the verification.
  4. Go back to the search engine and click Verify: the tag is already in the head of the community's pages. Then submit the sitemap as in the steps below.

Through the assistant, ask "Add this Google verification code to the community": update_community_settings takes google_site_verification and bing_site_verification (the code or the whole <meta> tag), and get_community_settings shows both. Owner and admin only.

Step by step: Google Search Console

Search Console shows which pages Google indexed, the searches that bring people in, and errors. Verification needs DNS access, so it applies to a custom domain; on the Memberfy address, Google finds the sitemap on its own through robots.txt.

Role: whoever manages the domain's DNS.

  1. Open search.google.com/search-console and click Add property.
  2. Choose Domain and type the main domain, without https:// and without the subdomain: papodesysadmin.org.
  3. Google shows a TXT record. Create it in your DNS provider's panel (Cloudflare, GoDaddy, Namecheap…), under the name @ (or the root domain), and click Verify. DNS can take a few minutes.
  4. In the left menu, open Sitemaps, type the full address, https://community.papodesysadmin.org/sitemap.xml, and click Submit.
  5. Come back in a few days: Pages shows what was indexed and why each page was left out.

Step by step: Bing Webmaster Tools

Bing also feeds other search engines and assistants that use its index.

  1. Open bing.com/webmasters and sign in with a Microsoft, Google or Facebook account.
  2. The shortest path is Import from Google Search Console: it brings the site and the sitemap already verified. To do it by hand, add https://community.papodesysadmin.org and choose verification by DNS record (CNAME).
  3. Under Sitemaps, click Submit sitemap and paste https://community.papodesysadmin.org/sitemap.xml.

Examples

Excel School, on school.excelpractice.com. The Start here section is Public, with the Welcome and Free lessons spaces; Student area is Private with the Complete plan, with a padlock; March class is Private for the class group only. Result:

  • The sitemap has the home page, Welcome, Free lessons, the upcoming events from the open calendar and /pricing.
  • llms.txt lists the two open spaces, the events, the Complete plan with its monthly and yearly price and, under Requires sign-in or purchase, Student area — requires a plan. March class doesn't appear.

Tips

  • Write the Community Description even if you fill in the SEO Description: the first one also shows inside the community.
  • Give every open space a description: it becomes the page's description on Google and the space's line in llms.txt.
  • Use a padlock (not Hide) on the paid areas you want to sell: the name and description reach AI with the way in, without revealing the content.
  • With a custom domain, submit the custom domain's sitemap, not the Memberfy address's: that's the one the pages point to as the official address.

Common errors and how to fix them

SituationWhat to do
A space isn't in the sitemapCheck that it's Public, and that its section is too: a space is never more open than its section
Search shows "a community on Memberfy" as the descriptionFill in the Community Description or the SEO Description and wait for Google to reread the page
Search Console says "Blocked by robots.txt" on a pageIt's an administration, checkout or account page: it shouldn't be indexed
Search Console says "Excluded by 'noindex' tag"It's a padlocked space, a member page or an invite link: it's right to leave it out
Search Console says "Not found (404)"A section or space was deleted, renamed or closed. If you renamed it, the new address goes into the sitemap on its own
I can't verify the .memberfy.net addressUse verification by tag: paste the Google or Bing code in the community's search settings (see Verify the community)

FAQ

Can I turn off the sitemap or llms.txt? No. They only have what anyone already sees without joining the community. To take a space out, change its visibility or use Hide.

Does the content of posts and lessons go to AI? Not from closed areas: only the name and what it takes to get in. For Public spaces, the file has the name, the description and the link; an AI that follows the link reads the page like any visitor.

Can I write my own robots.txt? No. Memberfy's blocks only the private areas and allows the rest.

Are the prices in llms.txt the ones on the plans page? Yes, the same, in each option's currency. Link only plans aren't included.

In the API

  • PUT /api/communities/{id} accepts googleSiteVerification and bingSiteVerification (owner and admin): that engine's code or whole tag; only the code is kept, and empty or null clears it. Other text returns 400 with the field in errors[].param. Both come out in GET /api/communities/settings.
  • GET /api/public/community-summary (no session; the community comes from X-CommunityId): courses, callsForPapers (with closesAt) and certificates, each item with title, description (one line, or null), path and updatedAt. Up to 200 per list, newest first; cached for 5 minutes and up to 120 reads per minute per IP. It never carries the certificate holder's name or e-mail. With Guest Access off (the default), the three lists come back empty, indexable certificates included.