Oğuzhan KARAGÜZEL

OĞUZHAN KARAGÜZEL

Software Developer

Technical SEO: robots.txt, Sitemap and Canonical Guide

Author: Oğuzhan KARAGÜZEL ·

This is the second article in the SEO series. In the first part we covered the basics of on-page SEO (title, meta description, headings and slugs). Now we move on to the technical layer that talks directly to search engine bots: robots.txt, sitemap.xml, canonical, hreflang and security settings.

The examples below lean on ASP.NET Core, but the logic is the same whatever stack you use. In the next part we will look at Open Graph, structured data and ARIA.

What Is Technical SEO?

Technical SEO is everything that makes a site discoverable, crawlable and indexable by search engines, independent of the content itself. However good your content is, it cannot rank if a bot cannot reach the page or cannot tell which URL is the real one.

This layer is largely the developer's responsibility. In this article we cover:

  • robots.txt: tells bots which areas they may crawl.
  • sitemap.xml: provides a map of the pages on your site.
  • Canonical, alternate and hreflang: manage duplicate content and multilingual structure.
  • Security: HTTPS redirects and form protection.

What Is robots.txt and What Is It For?

robots.txt is a plain text file that sits in the root of your site and tells search engine crawlers which sections they may or may not crawl. When a bot arrives at your site, the first thing it requests is yourdomain.com/robots.txt. You can open the robots.txt of any site in your browser and study it.

  • Managing crawl budget: Keep crawlers away from areas you do not want crawled, such as admin panels, internal search results and filter combinations.
  • Reducing server load: Stop bots from needlessly straining your server.
  • Announcing the sitemap: Tell bots where your sitemap lives.

Important: robots.txt is not a security mechanism. Anyone can read the file, and obeying it is up to the bot. A page blocked with Disallow can even still show up in search results if other sites link to it. Protect truly private content with authentication, and use the noindex meta tag for pages you want removed from results (do not block those pages in robots.txt, otherwise the bot cannot see the tag).

How to Write a robots.txt File

The file is built from four basic directives:

  1. User-agent: Names the bot the rule applies to. * means all bots.
  2. Disallow: Blocks the given path from being crawled.
  3. Allow: Permits a specific path inside a blocked directory.
  4. Sitemap: Gives the full address of your sitemap file.

A typical example:

robots.txt
User-agent: *
Disallow: /admin/
Disallow: /login/
Allow: /public/

Sitemap: https://www.example-site.com/sitemap.xml

This file tells all bots not to crawl the /admin/ and /login/ directories, allows /public/, and announces the sitemap. Always write the sitemap address as a full URL, including https.

Managing robots.txt in ASP.NET

In ASP.NET Core, dropping a robots.txt file into wwwroot is enough for most projects. Two points deserve attention, though:

  • Different content per environment: Test and development environments must not appear in search engines, so return Disallow: / there. In production, restrict paths that should not be crawled (such as the admin area) and allow everything else.
  • Know what you return: a 404 for /robots.txt is not a problem; Google reads it as "everything is allowed". The real danger is a 5xx error: if robots.txt returns a server error, Google may stop crawling the site for a while. That is why it is best to answer the request deliberately in every environment.

A practical way to satisfy both needs is to handle the request in middleware and read files such as Robots.Development.txt and Robots.Production.txt depending on the environment:

C#
app.Use(async (context, next) =>
{
    if (!context.Request.Path.Equals("/robots.txt", StringComparison.OrdinalIgnoreCase))
    {
        await next();
        return;
    }

    var robotsPath = Path.Combine(
        app.Environment.ContentRootPath,
        $"Robots.{app.Environment.EnvironmentName}.txt");

    // Fall back to a safe default if the file does not exist
    var output = "User-agent: *\nAllow: /";

    if (File.Exists(robotsPath))
    {
        output = await File.ReadAllTextAsync(robotsPath);
    }

    context.Response.ContentType = "text/plain; charset=utf-8";
    await context.Response.WriteAsync(output);
});

What Is sitemap.xml and How Do You Write It?

sitemap.xml is an XML file that lists the pages of your site. It helps search engines find all your content and notice new or updated pages quickly. It matters most for large, deeply nested and multilingual sites.

The basic structure looks like this:

XML
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://www.example.com/</loc>
    <lastmod>2026-10-23</lastmod>
  </url>
  <url>
    <loc>https://www.example.com/articles/sample-post</loc>
    <lastmod>2026-10-20</lastmod>
  </url>
</urlset>
  • <url>: One entry per page.
  • <loc>: The full URL of the page (required).
  • <lastmod>: The date the page was last updated. Useful when kept accurate.
  • <changefreq> and <priority>: Common in older guides, but Google ignores both fields. You do not have to include them.

Sitemap Structure for Multilingual Sites

On multilingual or multi-regional sites you can build a hierarchy instead of a single file: the main file (/sitemap.xml) points to language-specific sitemaps (for example /tr/sitemap.xml and /en/sitemap.xml). Whether you use one file or a hierarchy, the rule stays the same: the first address search engines reach must be yourdomain.com/sitemap.xml, and it must be announced in robots.txt with a Sitemap: line.

The main file that points to other sitemaps uses the <sitemapindex> root element, not <urlset>:

XML
<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <sitemap>
    <loc>https://www.example.com/tr/sitemap.xml</loc>
    <lastmod>2026-10-23T11:30:18+03:00</lastmod>
  </sitemap>
  <sitemap>
    <loc>https://www.example.com/en/sitemap.xml</loc>
    <lastmod>2026-10-23T11:30:18+03:00</lastmod>
  </sitemap>
</sitemapindex>

You can place the language files physically under wwwroot as tr/sitemap.xml and en/sitemap.xml, but this becomes hard to manage as languages are added. A more flexible approach is to keep files such as sitemap.tr.xml and sitemap.en.xml and serve them from /{language}/sitemap.xml through middleware:

C#
using System.Text.RegularExpressions;

app.Use(async (context, next) =>
{
    // /tr/sitemap.xml  or  /en/sitemap.xml/
    var path = context.Request.Path.Value?.TrimEnd('/') ?? string.Empty;
    var match = Regex.Match(path, @"^/(?<culture>[a-z]{2})/sitemap\.xml$", RegexOptions.IgnoreCase);

    if (!match.Success)
    {
        await next();
        return;
    }

    var culture = match.Groups["culture"].Value.ToLowerInvariant();
    var file = Path.Combine(app.Environment.ContentRootPath, $"sitemap.{culture}.xml");

    if (!File.Exists(file))
    {
        context.Response.StatusCode = StatusCodes.Status404NotFound;
        return;
    }

    context.Response.ContentType = "application/xml; charset=utf-8";
    await context.Response.WriteAsync(await File.ReadAllTextAsync(file));
});

Once the sitemap is ready, do not forget to submit it in Google Search Console.

Canonical URLs

A canonical URL is a link element that tells search engines which address is the original (canonical) version when the same or very similar content exists at more than one address. The tag goes in the <head> of the page:

HTML
<head>
  <link rel="canonical" href="https://www.example.com/blog/post-title" />
</head>

The canonical tag solves four important problems:

  • Duplicate content: Search engines cannot tell which URL to rank; canonical removes the ambiguity.
  • Ranking performance: Strength is consolidated on one address instead of being split.
  • Crawl budget: Bots spend time on important pages instead of crawling the same content repeatedly.
  • Backlink value: The value of links pointing to the same content (often called link juice) is consolidated on a single URL.

The most common situations are product pages that multiply with options such as color and size, URL parameters such as ?utm_source=... or ?sort=price, and different URL structures serving the same content.

Things to Keep in Mind

  • Every page should contain a canonical pointing either to itself or to the original page (a self-referencing canonical is valid).
  • Always use a full URL, and keep the protocol (https) and the www choice consistent with the rest of the site.
  • Use canonical addresses in internal links as well.

On the developer side, the job is to write the canonical address correctly into every page's <head>. If the editor has set a canonical, use that address; if not, use the page's own clean address (without query parameters).

An Example Scenario

Suppose a health website has an article live at /what-helps-a-cold. To attract more search traffic, the editor wants to publish the same article at /common-cold and /what-is-a-common-cold as well. Since all three pages carry identical content, all three should point to /what-helps-a-cold as their canonical.

Two details matter here. First, which page is canonical, and whether a page is marked at all, should be left to the content editor: a keyword-heavy page published only for a short time and removed later may not need a canonical. That is why the canonical field should be editable in the CMS. Second, where possible it is cleaner to 301-redirect the extra addresses to the main one instead of duplicating the content.

Alternate, hreflang and x-default

On multilingual sites every page should declare its language versions with rel="alternate" and hreflang. That way search engines show the Turkish version to Turkish users and the English version to English users. The rules:

  • Every language version must list all versions, itself included (the links must be reciprocal).
  • Addresses must be full URLs.
  • x-default marks the default page for users who match no specific language or region.
HTML
<head>
  <link rel="alternate" hreflang="tr" href="https://www.example.com/tr/ornek-sayfa" />
  <link rel="alternate" hreflang="en" href="https://www.example.com/en/example-page" />
  <link rel="alternate" hreflang="x-default" href="https://www.example.com/en/example-page" />
</head>

Outdated tags: Older guides describe rel="alternate" media="..." for mobile versions and rel="prev" / rel="next" for paginated content. If you use responsive design (and do not run a separate mobile subdomain), you do not need the mobile alternate tag. Google has also said it no longer uses prev/next for ranking; for paginated lists a self-referencing canonical on every page and normal crawlable links are enough.

To verify your canonical and alternate tags you can use Google Search Console, Screaming Frog, or tools such as Ahrefs and Semrush.

Security: HTTPS Redirects and Captcha

Security is part of SEO too, but two basic rules cover most cases:

  1. Protect public forms. Contact, registration and application forms are the first target of spam bots; use a captcha or similar protection. Spam hurts both user experience and site reputation.
  2. Redirect http requests to https with a 301. A permanent (301) redirect passes ranking value to the https address and prevents the site from being indexed as two separate copies, http and https.

To make the redirect status code 301 in ASP.NET Core:

C#
builder.Services.AddHttpsRedirection(options =>
{
    options.RedirectStatusCode = StatusCodes.Status301MovedPermanently;
});

var app = builder.Build();

app.UseHttpsRedirection();

Alternatively, the URL Rewrite middleware gives the same result:

C#
var rewriteOptions = new RewriteOptions()
    .AddRedirectToHttpsPermanent();

app.UseRewriter(rewriteOptions);

In production you can also add the HSTS header with app.UseHsts(), which stops browsers from ever connecting to the site over http again.

Technical SEO Checklist

Run through this list quickly before going live:

  1. Does /robots.txt open, and does it avoid blocking the whole site in production by mistake?
  2. Are test and development environments closed to search engines?
  3. Does /sitemap.xml exist, is it announced in robots.txt, and has it been submitted to Search Console?
  4. On multilingual sites, are the language sitemaps and the sitemap index correct?
  5. Does every page have a correct canonical tag?
  6. Are reciprocal hreflang tags and x-default defined on multilingual pages?
  7. Are http requests redirected to https with a 301?
  8. Are public forms protected against spam?

In the next part of the series we move on to Open Graph, structured data (Schema.org) and ARIA.