Analytics, SEO and growth

Most tools work through a small script on every page. Each visit records a page view with details such as the referring site, campaign tags, country, device and browser, plus custom events such as sign-ups, button clicks and purchases. Dashboards turn that into visitors, sessions, top pages, traffic sources, conversion rates and funnels, which show how many people get through each step of, say, a checkout.

The main families are Google Analytics 4 (free, detailed and tied to Google Ads), cookieless tools such as Plausible, Fathom and Umami (simple and privacy-first), and product analytics such as PostHog, Mixpanel and Amplitude, which follow individual users through an app. Server logs and a CDN's own statistics count requests without any script at all.

No tool sees everyone. Ad blockers, declined cookie banners and bots all skew the numbers, so two tools rarely agree exactly. Trends over weeks and months say more than any single figure.

A Google tag (gtag.js, or a tag in Google Tag Manager) on each page sends events to a GA4 property. Page views, scrolls, outbound clicks, site searches and file downloads are collected automatically, and you add your own, such as sign_up, or purchase with a value. The events that matter most are marked as key events (formerly called conversions). Standard reports cover traffic, engagement and revenue; Explorations build funnels, paths and segments; and a free BigQuery export hands over the raw events.

GA4 replaced Universal Analytics, which stopped collecting data in 2023, and its event-based model takes some learning. The free tier samples very large Explorations, hides small numbers when some privacy features are on, and keeps user-level data for Explorations for 14 months at most. Analytics 360, the enterprise tier, raises those limits and adds service agreements.

It stores first-party cookies (_ga) and sends data to Google, so in the EU, the UK and similar places it normally needs consent through a cookie banner. Consent Mode tells Google tags what each visitor agreed to, and Google models some of the missing data. Ad blockers stop it for part of the audience.

You install one container snippet on the site. In the GTM dashboard you then define tags (the scripts to run, such as GA4, Google Ads, the Meta Pixel or LinkedIn's Insight Tag), triggers (when to run them: page views, clicks, form submissions, custom events) and variables. The site's own code can push events and details, such as an order total, into a data layer that tags read. Changes are published as versions, with a preview mode for testing first.

It frees marketers from waiting on developers, but every tag adds third-party JavaScript that can slow pages, leak data or break things, so access and reviews matter. It works with Consent Mode and cookie banners to hold tags back until a visitor agrees. Server-side tagging runs a container on a server you control, which receives events from the site and forwards them to each vendor, so fewer scripts load in the browser and you decide what is shared.

Tools such as Plausible, Fathom, Umami, Simple Analytics and Cloudflare Web Analytics record page views, referrers, campaigns, countries, devices and custom goals without storing anything on the visitor's device. To count unique visitors, some combine the IP address and browser details with a secret that changes every day and keep only the resulting hash, so nobody can be followed from one day, or one site, to the next.

The scripts are a small fraction of the size of Google's tag, and the dashboards fit on one page. The trade-off is depth: there are no long-term user journeys, fewer audience features and weaker ad attribution. Because nothing is stored on the device and no personal data is kept, many sites run them without a consent banner, although the legal position depends on the country and on the setup.

Plausible (AGPL) and Umami (MIT) are open source and can run on your own server or be used as hosted cloud services; Fathom is a hosted service only.

There are five classic tags: utm_source (who sent the visit, such as google or newsletter), utm_medium (the channel, such as cpc, email or social), utm_campaign (the campaign name), and the optional utm_term and utm_content for keywords and ad variants. Analytics tools read them from the landing page's address and credit the visit, and any later conversion, to that campaign. The name comes from Urchin, the analytics company Google bought and turned into Google Analytics.

Consistent naming matters: GA4 treats Email and email as different sources, so teams keep a shared naming sheet or a link builder. Tagging internal links on your own site is a mistake, because it overwrites where the visitor originally came from. Ad platforms add their own click ids automatically (gclid for Google Ads, fbclid for Meta), and canonical tags stop tagged URLs from showing up as duplicate pages in search.

The base code loads Meta's script and sends a PageView on every page; you add standard events such as ViewContent, AddToCart, Lead and Purchase (with value and currency), or custom events of your own. It sets a first-party _fbp cookie and keeps the fbclid id from ad clicks, so Meta can match events to its users, credit ads with the conversions they drove, build audiences such as people who viewed a product, and steer ad delivery towards people likely to act.

Because it runs in the browser, ad blockers, Safari and Firefox tracking protection and visitors who decline cookies all cut into the data, and under GDPR and similar laws it should only load after consent. Meta recommends pairing it with the Conversions API and deduplicating the two with a shared event id. Events Manager shows what arrives, and the Meta Pixel Helper browser extension helps debug a setup.

When something that matters happens, such as an order being paid, your backend sends an HTTPS request to Meta's Graph API with an access token, the event name and time, the page URL and customer details. Email addresses and phone numbers are normalised and hashed with SHA-256 before sending; the _fbp and _fbc cookie values, IP address and browser user agent go along unhashed to help matching. Events Manager grades the result with an Event Match Quality score.

Because it runs on your server, it also reports events a browser never sees: a payment confirmed later by a webhook, a phone order, an in-store sale or a lead that becomes a customer in your CRM. When the Pixel and the API both send the same event, giving both the same event name and event id lets Meta keep one copy, as long as they arrive within 48 hours of each other.

Setup can be direct code, a platform integration (Shopify, WooCommerce), server-side Google Tag Manager or Meta's Conversions API Gateway. TikTok, Google Ads, LinkedIn and Snap offer similar server-side APIs.

Search engines work in three steps. They crawl the web by following links and sitemaps, index what they find (store it and work out what each page is about), and rank indexed pages for each search using many signals: relevance to the query, content quality, links from other sites, freshness, location and page experience. SEO is work on each of those steps.

Technical SEO keeps a site crawlable and fast: clean URLs, HTML that arrives already rendered, a sitemap and robots.txt, canonical tags, structured data, mobile-friendly layouts and good Core Web Vitals. On-page SEO is the content itself: pages that answer what people actually search for, with clear titles, meta descriptions, headings and internal links. Off-page SEO is reputation: links and mentions from other sites, reviews and, for local businesses, a Google Business Profile.

Results take weeks to months, nobody can guarantee a ranking, and shortcuts such as bought links or mass-produced pages break Google's spam policies. AI answers draw on the same search index, so the same groundwork also supports answer engine optimisation.

AI answer engines usually run a web search, read the top results and write a summary with links to the pages they used. Being one of those sources rests mostly on the same things as SEO: pages that can be crawled and indexed, and content that is clear, specific and trustworthy. Google states that optimising for its AI Overviews and AI Mode is still SEO, and that no special files or markup are needed.

Habits that help: answer the question directly near the top of the page, back claims with specific facts and sources, keep pages up to date, earn mentions on other reputable sites, and serve content as HTML rather than only through JavaScript, since many AI crawlers do not run scripts. Search crawlers such as OAI-SearchBot (for ChatGPT) and PerplexityBot need to be allowed in robots.txt for a site to appear in those tools.

Measuring it is still rough. Search Console shows impressions in Google's AI features, and analytics tools show referrals from sites such as chatgpt.com and perplexity.ai. Promises of guaranteed placement in AI answers deserve caution.

After you prove you own a site (with a DNS record, an HTML file, a meta tag, or through Analytics or Tag Manager), the Performance report lists search queries with clicks, impressions, click-through rate and average position for the last 16 months. The Page indexing report explains why pages are or are not in Google's index, URL Inspection shows how Google sees a single page and lets you request indexing, and sitemaps are submitted here too.

Other reports cover Core Web Vitals from real Chrome users, structured data errors, manual penalties, security problems and links. Newer reports show impressions in generative AI features such as AI Overviews and AI Mode. Bing Webmaster Tools is the equivalent for Bing and can import a Search Console setup.

The vocabulary is schema.org, backed by Google, Microsoft, Yahoo and Yandex, with types such as Organization, Product, Article, Event, Recipe, LocalBusiness and BreadcrumbList. JSON-LD places it in a script tag of type application/ld+json as a block of JSON, separate from the visible HTML, and is the format Google recommends; Microdata and RDFa weave the same information into HTML attributes instead.

Valid markup makes a page eligible for rich results such as star ratings, prices, event dates and breadcrumbs, and helps search engines and AI systems read facts reliably. It is not a ranking boost on its own, it must match what visitors can see, and Google has retired several rich result types over the years (How-to, then FAQ), so check the current list. Google's Rich Results Test and the Schema Markup Validator check the code.

Facebook introduced the Open Graph protocol in 2010, and nearly every app that builds link previews now reads it. The core tags are og:title, og:description, og:image, og:url and og:type, written as meta tags in the page's head. X reads its own twitter:card tags and falls back to Open Graph for the rest. An image of 1200 by 630 pixels displays well on most platforms.

Previews are fetched once and cached, so an updated image may not appear until the cache is refreshed with tools such as Facebook's Sharing Debugger or LinkedIn's Post Inspector. Image URLs must be absolute and publicly reachable. Frameworks such as Next.js can generate a card image for every page automatically.

The same content often lives at several addresses: with and without www, over http and https, with tracking parameters such as ?utm_source=, with sort or filter options, or as a copy republished on another site. A link element with rel='canonical' in the page's head (or a Link HTTP header, for files such as PDFs) names the preferred version, so ranking signals gather on that URL and it is the one shown in results.

Google treats it as a strong hint rather than a command, weighing it alongside redirects, sitemaps and internal links, and URL Inspection in Search Console shows which version it chose. Use absolute URLs, point an original page at itself, and use a permanent (301) redirect instead when the duplicate does not need to exist at all.

A sitemap is an XML file, usually at /sitemap.xml, holding a url entry for each page with its address (loc) and, ideally, its last modified date (lastmod). The format, published at sitemaps.org, is shared by Google, Bing and most other engines. One file can hold up to 50,000 URLs or 50 MB uncompressed; bigger sites split into several files listed in a sitemap index, and extensions cover images, video and news.

Submit it in Search Console and Bing Webmaster Tools, or point to it with a Sitemap line in robots.txt. It helps pages get discovered but does not force them into the index. Google ignores the priority and changefreq fields and trusts lastmod only when it is consistently accurate. Most frameworks and CMSs generate one automatically.

It lives at /robots.txt and groups rules by crawler: a User-agent line (Googlebot, Bingbot, GPTBot, or * for everyone) followed by Allow and Disallow lines with paths, plus an optional Sitemap line. The format was standardised as RFC 9309 in 2022. Reputable crawlers obey it, but it is a request, not a lock: it offers no security, and anyone can read the file.

It controls crawling, not indexing. A blocked page can still appear in results, without a description, if other sites link to it; to keep a page out of search, let it be crawled and add a noindex robots meta tag or X-Robots-Tag header. AI companies often use separate crawlers for training and for search (OpenAI has GPTBot and OAI-SearchBot), so each can be allowed or blocked on its own.

You generate a key, host it as a text file on your site to prove ownership, and then send each changed URL, or up to 10,000 of them in one JSON request, to an IndexNow endpoint. Participating engines share submissions with each other, so one ping reaches Bing, Yandex, Naver, Seznam.cz, Yep and the rest. A success response only means the URL was received, not that it will be indexed.

Microsoft Bing and Yandex launched it in 2021. Google does not take part, so sitemaps and Search Console remain the way to reach it. Many CMS plugins, hosting platforms and CDNs (Cloudflare's Crawler Hints, for example) send pings automatically, and it suits sites whose pages change often, such as shops, listings and news.

Jeremy Howard of Answer.AI proposed it in 2024. The file opens with the site's name as a heading and a one-line summary, followed by sections of links with short notes, ideally pointing to clean Markdown versions of the pages. It targets AI agents and coding assistants that need concise context rather than pages full of navigation, adverts and scripts. Some sites also publish an llms-full.txt with the complete text.

Support is uneven. It is most common on software documentation, including the developer docs of several AI companies, and Chrome's Lighthouse checks for it in its Agentic Browsing audits, but Google says its Search, AI Overviews included, ignores the file. It does not control access either; that remains the job of robots.txt.

Largest Contentful Paint (LCP) is the time until the biggest image or block of text appears; good is 2.5 seconds or less. Interaction to Next Paint (INP), which replaced First Input Delay in 2024, measures how long the page takes to respond visibly to taps, clicks and key presses; good is 200 milliseconds or less. Cumulative Layout Shift (CLS) scores how much content moves unexpectedly as the page loads; good is 0.1 or less.

A page passes when at least 75 per cent of real visits meet all three targets, measured from Chrome users over the previous 28 days in the Chrome UX Report. That field data appears in Search Console and PageSpeed Insights, while lab tools such as Lighthouse simulate one load to help diagnose problems. The scores feed Google's page experience signals, which weigh less than relevance, but slow, jumpy pages lose visitors either way.

Lighthouse runs in Chrome DevTools, from the command line, as a Node module, in CI pipelines through Lighthouse CI, and behind PageSpeed Insights. It loads the page under controlled lab conditions, by default a simulated mid-range phone on a slow mobile connection, scores each category from 0 to 100, and lists audits such as oversized images, render-blocking scripts or missing alt text, each with an explanation.

Lab scores vary between runs and machines, and they are not what Google uses in search; that is field data from real Chrome users (Core Web Vitals). Treat the score as a diagnostic rather than a goal. Recent versions add an Agentic Browsing category that checks how easily AI agents can read and use a page, including a check for llms.txt.

Images are usually the heaviest part of a page. The main techniques are modern formats (WebP and AVIF files are typically much smaller than JPEG or PNG at similar quality), resizing to the displayed size with srcset and sizes so phones never download desktop-sized images, sensible compression, lazy loading for images below the fold with loading='lazy', and SVG for logos and icons.

Setting width and height, or an aspect ratio, reserves space and prevents layout shift. The main hero image is often the page's LCP element, so it should load early rather than lazily, often with fetchpriority='high'. Much of this can be automated: next/image in Next.js, Astro's image component, image CDNs such as Cloudinary, imgix or Cloudflare Images, and libraries such as sharp in a build step.