Building a Running Events Site with PHP and React, Part 3: Scraping Running Events from the Web

So far our races come from a seed script. Today we replace typing with collecting: a scraper that reads two Slovenian race calendars, turns their pages into clean event data, and saves it through the same insert-or-update logic as Part 2. It runs once a day in its own container.

As before, every file is listed in full with its path above it.

The Series

Part	Topic
1	A Dockerized Laminas API
2	Modeling events with Doctrine ORM
3	Scraping running events from the web (this article)
4	Documenting the API with OpenAPI
5	A React frontend with type-safe API calls

Look Before You Scrape

The most common scraping mistake happens before any code is written: assuming the data you see in the browser is in the HTML the server sends. Check first. Our main source is tekaški-koledar.si, a calendar of Slovenian running races. Its search page shows hundreds of races in the browser, but:

curl -s https://tekaski-koledar.si/iskalnik-tekov/ | grep -c "maraton"

prints 0. Not a single marathon in the HTML. The page is built with JavaScript, which loads the races after the page opens. A PHP scraper doesn't run JavaScript, so it sees an empty list.

Two places are worth checking whenever that happens:

The sitemap. Sites publish /sitemap.xml so search engines can find every page. tekaški-koledar.si lists about 300 individual race pages there, under /tek/2026/..., and those pages are plain HTML. The same grep against https://tekaski-koledar.si/sitemap.xml finds dozens of marathons.
Structured data. Many sites embed a machine-readable description of the page for search engines, as JSON inside a <script type="application/ld+json"> tag, using the schema.org vocabulary. Look at the source of one race page and you'll find:

{
  "@context": "https://schema.org",
  "@type": "Event",
  "name": "Zimski trail tek Bohinj",
  "startDate": "2026-03-07",
  "location": {"@type": "Place", "name": "Bohinj", "address": "Bohinj, Gorenjska, Slovenija"},
  "description": "Zimski trail tek Bohinj – 7. marec 2026, Bohinj. Razdalje: 6 km · 12 km · 18 km. ..."
}

This is the best case for a scraper. The date is already in ISO format, so there's no parsing of "7. marec 2026". And unlike CSS classes, which change whenever a site is redesigned, structured data is maintained for Google, so site owners have a strong reason to keep it stable.

Our second source, rekreacija.si, works differently, which is exactly why it's useful here:

	tekaški-koledar.si	rekreacija.si
Finding events	sitemap	paginated "running" category pages
Event data	schema.org `Event`	schema.org `SportsEvent`
Region	one part of an address string	an `addressRegion` field, or appended to the locality
Distances	listed in the description	only mentioned in free text
Quirks	one region uses a short name	times labeled UTC that are really local; a race in Croatia marked as Slovenian

Each source gets its own adapter, a class that knows that one site's structure. Everything else (fetching politely, handling errors, saving to the database) is shared.

Scraping Politely

A scraper uses someone else's server. Ours follows a few simple rules:

Check robots.txt. Both sites allow crawling.
Identify yourself. Every request sends a User-Agent with a contact address, so a site owner can see who you are and reach you.
Go slowly. One request per second, and one full run per day. That's about 350 requests a day, a tiny load for any website.
Fetch only what you need. Past years' races are skipped before they're downloaded.

Step 1: Install the Packages

docker compose run --rm php composer require --no-interaction laminas/laminas-cli symfony/http-client symfony/dom-crawler symfony/css-selector

laminas-cli runs command-line commands inside our Mezzio application, with the same service container and configuration as the API.
symfony/http-client makes HTTP requests.
symfony/dom-crawler with symfony/css-selector finds elements in HTML using CSS selectors like article h3 a.

(laminas-cli is already present as a dependency of the development tooling, but the scraper is production code, so we require it directly.)

Then create a module for the scraper, separate from the Events module that owns the data:

docker compose run --rm php composer mezzio mezzio:module:create Scraper

All files below go inside api/src/Scraper/src/ unless stated otherwise.

Step 2: The Building Blocks

An adapter produces ScrapedEvent objects: plain data, independent of Doctrine, so the parsing code can be tested without a database.

api/src/Scraper/src/ScrapedEvent.php

<?php

declare(strict_types=1);

namespace Scraper;

use DateTimeImmutable;

final readonly class ScrapedEvent
{
    /**
     * @param list<int> $distances Distances in meters.
     */
    public function __construct(
        public string $externalId,
        public string $name,
        public DateTimeImmutable $date,
        public string $place,
        public ?string $region,
        public string $url,
        public array $distances,
    ) {
    }
}

Every adapter implements this interface. It has two jobs: find the URLs of event pages, and turn one page's HTML into a ScrapedEvent, or null to skip it.

api/src/Scraper/src/EventSource.php

<?php

declare(strict_types=1);

namespace Scraper;

interface EventSource
{
    public function name(): string;

    public function baseUrl(): string;

    /**
     * Finds the URLs of individual event pages.
     *
     * @return list<string>
     */
    public function eventUrls(): array;

    /**
     * Turns one event page into an event, or returns null to skip it.
     */
    public function parse(string $url, string $html): ?ScrapedEvent;
}

Keeping parse() free of HTTP is a deliberate choice: it takes HTML in and returns data out, so we can test it with saved pages (Step 7).

All requests go through one small class that enforces the delay between requests:

api/src/Scraper/src/PageFetcher.php

<?php

declare(strict_types=1);

namespace Scraper;

use Symfony\Contracts\HttpClient\HttpClientInterface;

final class PageFetcher
{
    private float $lastRequestAt = 0.0;

    public function __construct(
        private readonly HttpClientInterface $http,
        private readonly int $delayMs,
    ) {
    }

    public function get(string $url): string
    {
        $wait = $this->lastRequestAt + $this->delayMs / 1000 - microtime(true);
        if ($wait > 0) {
            usleep((int) ($wait * 1_000_000));
        }
        $this->lastRequestAt = microtime(true);

        // getContent() throws for 4xx and 5xx responses.
        return $this->http->request('GET', $url)->getContent();
    }
}

Step 3: The Parsers

Three small helpers are shared by both adapters. The first finds the schema.org event in a page. Sites embed structured data in a few shapes (a single object, a list, or a @graph of several objects), and the event may be an Event or a subtype like SportsEvent. This handles all of them:

api/src/Scraper/src/Parser/JsonLd.php

<?php

declare(strict_types=1);

namespace Scraper\Parser;

use Symfony\Component\DomCrawler\Crawler;

final class JsonLd
{
    /**
     * Returns the first schema.org Event (or subtype such as SportsEvent) on the page.
     *
     * @return array<string, mixed>|null
     */
    public static function findEvent(string $html): ?array
    {
        $blocks = (new Crawler($html))
            ->filter('script[type="application/ld+json"]')
            ->each(static fn (Crawler $node): string => $node->text(normalizeWhitespace: false));

        foreach ($blocks as $json) {
            $data = json_decode($json, true);
            if (! is_array($data)) {
                continue;
            }

            $items = $data['@graph'] ?? (array_is_list($data) ? $data : [$data]);
            foreach ($items as $item) {
                if (is_array($item) && str_ends_with((string) ($item['@type'] ?? ''), 'Event')) {
                    return $item;
                }
            }
        }

        return null;
    }
}

The second turns distance text into meters. Slovenian sites write decimals with a comma ("21,1 km"), mix kilometers and meters ("800 m" for children's races), and sometimes use words ("polmaraton" is a half marathon).

api/src/Scraper/src/Parser/DistanceParser.php

<?php

declare(strict_types=1);

namespace Scraper\Parser;

final class DistanceParser
{
    /**
     * Extracts distances in meters from text such as "6 km · 21,1 km · 800 m".
     *
     * @return list<int>
     */
    public static function parse(string $text): array
    {
        $meters = [];

        preg_match_all('/(\d+(?:[.,]\d+)?)\s*(km|m)\b/iu', $text, $matches, PREG_SET_ORDER);
        foreach ($matches as [, $number, $unit]) {
            $value    = (float) str_replace(',', '.', $number);
            $meters[] = (int) round(strtolower($unit) === 'km' ? $value * 1000 : $value);
        }

        if (preg_match('/\bpolmaraton/iu', $text)) {
            $meters[] = 21097;
        }
        if (preg_match('/\bmaraton/iu', $text)) {
            $meters[] = 42195;
        }

        $meters = array_values(array_unique(array_filter($meters, static fn (int $m): bool => $m > 0)));
        rsort($meters);

        return $meters;
    }
}

Note that "21,1 km" becomes 21100, not the official half-marathon distance of 21097. That's fine: Part 2's distance filter uses ranges, so minDistance=20000&maxDistance=22000 still finds it.

The third maps region names onto Slovenia's 12 official statistical regions. tekaški-koledar.si writes "Jugovzhodna" where the official name, and our seed data, says "Jugovzhodna Slovenija". Without normalizing, a region filter would split those races into two groups.

api/src/Scraper/src/Parser/Region.php

<?php

declare(strict_types=1);

namespace Scraper\Parser;

final class Region
{
    private const OFFICIAL = [
        'Pomurska', 'Podravska', 'Koroška', 'Savinjska', 'Zasavska', 'Posavska',
        'Jugovzhodna Slovenija', 'Osrednjeslovenska', 'Gorenjska',
        'Primorsko-notranjska', 'Goriška', 'Obalno-kraška',
    ];

    private const ALIASES = [
        'Jugovzhodna' => 'Jugovzhodna Slovenija',
    ];

    /**
     * Maps a region name onto one of Slovenia's 12 official regions, or null if it isn't one.
     */
    public static function normalize(?string $name): ?string
    {
        $name = trim((string) $name);
        $name = self::ALIASES[$name] ?? $name;

        return in_array($name, self::OFFICIAL, true) ? $name : null;
    }
}

Returning null for anything that isn't a Slovenian region turns out to be useful in the second adapter.

Step 4: The tekaški-koledar.si Adapter

api/src/Scraper/src/Source/TekaskiKoledarSource.php

<?php

declare(strict_types=1);

namespace Scraper\Source;

use DateTimeImmutable;
use Scraper\EventSource;
use Scraper\PageFetcher;
use Scraper\Parser\DistanceParser;
use Scraper\Parser\JsonLd;
use Scraper\Parser\Region;
use Scraper\ScrapedEvent;

/**
 * tekaski-koledar.si: race pages are listed in the sitemap, and each page
 * carries a schema.org Event with the address and a "Razdalje: ..." description.
 */
final class TekaskiKoledarSource implements EventSource
{
    private const BASE_URL = 'https://tekaski-koledar.si';

    public function __construct(private readonly PageFetcher $fetcher)
    {
    }

    public function name(): string
    {
        return 'tekaski-koledar.si';
    }

    public function baseUrl(): string
    {
        return self::BASE_URL;
    }

    public function eventUrls(): array
    {
        $sitemap = simplexml_load_string($this->fetcher->get(self::BASE_URL . '/sitemap.xml'));
        $year    = (int) date('Y');
        $urls    = [];

        foreach ($sitemap->url as $entry) {
            $url = (string) $entry->loc;
            // Race pages look like /tek/2026/r000123-name/; skip past years.
            if (preg_match('#/tek/(\d{4})/#', $url, $m) && (int) $m[1] >= $year) {
                $urls[] = $url;
            }
        }

        return $urls;
    }

    public function parse(string $url, string $html): ?ScrapedEvent
    {
        $event = JsonLd::findEvent($html);
        $date  = DateTimeImmutable::createFromFormat('!Y-m-d', substr((string) ($event['startDate'] ?? ''), 0, 10));

        if ($event === null || empty($event['name']) || $date === false) {
            return null;
        }

        // "Stahovica, Gorenjska, Slovenija" or "Kamnik, Velika planina, Gorenjska, Slovenija":
        // the region is whichever part of the address is a Slovenian region.
        $address = array_map('trim', explode(',', (string) ($event['location']['address'] ?? '')));
        $region  = null;
        foreach ($address as $part) {
            $region ??= Region::normalize($part);
        }

        // "... Razdalje: 6 km · 12 km · 18 km. Podrobnosti ..."
        preg_match('/Razdalje:\s*(.+?)(?:\.\s|$)/u', (string) ($event['description'] ?? ''), $m);

        return new ScrapedEvent(
            externalId: preg_match('#/tek/(\d{4}/r\d+)#', $url, $id) ? $id[1] : $url,
            name: trim((string) $event['name']),
            date: $date,
            place: (string) ($event['location']['name'] ?? $address[0]),
            region: $region,
            url: $url,
            distances: DistanceParser::parse($m[1] ?? ''),
        );
    }
}

A few decisions in this class:

The external ID is the year plus the site's record number, for example 2026/r000123. The rest of the URL contains the race name, which organizers sometimes edit; the record number doesn't change. Including the year keeps next year's edition of the same race a separate event.
The region is whichever part of the address is a Slovenian region. Most addresses have three parts ("Stahovica, Gorenjska, Slovenija"), but some have four ("Kamnik, Velika planina, Gorenjska, Slovenija"). An earlier version of this adapter took the second part, which silently lost the region for every four-part address. Checking each part against the list of regions works for both.
Distances are read only from the part of the description after "Razdalje:" ("distances:"). Running the distance parser over the whole description would also pick up numbers that aren't race distances. When a race has no distances yet, the site says "Razdalje: niso dodane" ("not added"), and the parser returns an empty list.

Step 5: The rekreacija.si Adapter

This site has no useful sitemap for our purpose, but it has a category page listing running events, 10 per page. The adapter walks through those pages and collects the event links with a CSS selector:

api/src/Scraper/src/Source/RekreacijaSource.php

<?php

declare(strict_types=1);

namespace Scraper\Source;

use DateTimeImmutable;
use Scraper\EventSource;
use Scraper\PageFetcher;
use Scraper\Parser\JsonLd;
use Scraper\Parser\Region;
use Scraper\ScrapedEvent;
use Symfony\Component\DomCrawler\Crawler;
use Symfony\Contracts\HttpClient\Exception\ClientExceptionInterface;

/**
 * rekreacija.si: running events are listed on paginated category pages,
 * and each event page carries a schema.org SportsEvent with a structured address.
 * Distances are only mentioned in free text, so we don't import them.
 */
final class RekreacijaSource implements EventSource
{
    private const BASE_URL  = 'https://www.rekreacija.si';
    private const LIST_PATH = '/sport-dogodkov/tek/';
    private const MAX_PAGES = 20;

    public function __construct(private readonly PageFetcher $fetcher)
    {
    }

    public function name(): string
    {
        return 'rekreacija.si';
    }

    public function baseUrl(): string
    {
        return self::BASE_URL;
    }

    public function eventUrls(): array
    {
        $urls = [];

        for ($page = 1; $page <= self::MAX_PAGES; $page++) {
            $listUrl = self::BASE_URL . self::LIST_PATH . ($page > 1 ? "page/{$page}/" : '');

            try {
                $html = $this->fetcher->get($listUrl);
            } catch (ClientExceptionInterface) {
                break; // a 404 means we've gone past the last page
            }

            $links = (new Crawler($html))
                ->filter('article.rekol-row h3 a')
                ->each(static fn (Crawler $a): string => (string) $a->attr('href'));

            if ($links === []) {
                break;
            }
            array_push($urls, ...$links);
        }

        return array_values(array_unique($urls));
    }

    public function parse(string $url, string $html): ?ScrapedEvent
    {
        $event   = JsonLd::findEvent($html);
        $address = $event['location']['address'] ?? [];
        // startDate looks like "2026-10-10T07:30:00+00:00", but the time is local.
        // Only the date part is reliable.
        $date = DateTimeImmutable::createFromFormat('!Y-m-d', substr((string) ($event['startDate'] ?? ''), 0, 10));

        // The region usually has its own field, but for some events it's appended
        // to the locality instead: "Dragatuš, Jugovzhodna".
        $region   = Region::normalize($address['addressRegion'] ?? null);
        $locality = [];
        foreach (array_map('trim', explode(',', (string) ($address['addressLocality'] ?? ''))) as $part) {
            if (Region::normalize($part) !== null) {
                $region ??= Region::normalize($part);
            } elseif ($part !== '') {
                $locality[] = $part;
            }
        }

        // Skip incomplete events and events outside Slovenia. The country field
        // can't be trusted (a race in Croatia is marked "SI"), the region can.
        if ($event === null || empty($event['name']) || $date === false || $region === null) {
            return null;
        }

        return new ScrapedEvent(
            externalId: basename(parse_url($url, PHP_URL_PATH) ?: $url),
            name: html_entity_decode(trim((string) $event['name']), ENT_QUOTES | ENT_HTML5),
            date: $date,
            place: $locality !== [] ? implode(', ', $locality) : (string) ($event['location']['name'] ?? ''),
            region: $region,
            url: $url,
            distances: [],
        );
    }
}

Four real-world problems show up in this one class:

The region in the wrong field. Most events have an addressRegion, but events in one region have it appended to the town instead: "addressLocality": "Novo mesto, Jugovzhodna". The adapter checks both and keeps the rest of the locality as the place name. Our first version only read addressRegion and quietly skipped three Slovenian races, which we noticed only by comparing the listing pages with what ended up in the database.
Times labeled UTC that aren't. The start time arrives as 2026-10-10T07:30:00+00:00, but the race starts at 7:30 local time. Converting it to the Ljubljana time zone would move it to 9:30, and a race at 23:30 would move to the next day. We use only the date part, which is correct.
A wrong country. The running category also lists an event in Baška, on the Croatian island of Krk, and its structured data says "addressCountry": "SI". The country field can't be trusted, but the region can: Baška has no Slovenian region, so Region::normalize() returns null and the event is skipped.
Distances only in prose. Event pages mention distances in sentences like "Odrasli tečejo 10 km" ("adults run 10 km"), mixed with numbers like "2 km from the centre". We could try to guess, but a wrong distance is worse than a missing one: a filter for half marathons would show races that aren't. So this adapter imports no distances. The events still appear in date and region searches, and a future version could add distances from a better field if the site adds one.

That last point is a general rule for scrapers: import what the source states clearly, and leave the rest empty rather than invent it.

Step 6: Importing and the Command

The importer is shared by all sources. It asks an adapter for URLs, fetches each page politely, and saves the result with the same insert-or-update pattern as Part 2's seed script:

api/src/Scraper/src/EventImporter.php

<?php

declare(strict_types=1);

namespace Scraper;

use DateTimeImmutable;
use Doctrine\ORM\EntityManagerInterface;
use Events\Entity\Event;
use Events\Entity\Source;
use Throwable;

final class EventImporter
{
    public function __construct(
        private readonly EntityManagerInterface $em,
        private readonly PageFetcher $fetcher,
    ) {
    }

    /**
     * @return array{created: int, updated: int, skipped: int, failed: list<string>}
     */
    public function import(EventSource $source, int $limit = 0): array
    {
        $result = ['created' => 0, 'updated' => 0, 'skipped' => 0, 'failed' => []];

        $sourceEntity = $this->em->getRepository(Source::class)->findOneBy(['name' => $source->name()])
            ?? new Source($source->name(), $source->baseUrl());
        $this->em->persist($sourceEntity);

        $urls = $source->eventUrls();
        if ($limit > 0) {
            $urls = array_slice($urls, 0, $limit);
        }

        foreach ($urls as $url) {
            try {
                $scraped = $source->parse($url, $this->fetcher->get($url));
            } catch (Throwable $e) {
                $result['failed'][] = sprintf('%s: %s', $url, $e->getMessage());
                continue;
            }

            if ($scraped === null) {
                $result['skipped']++;
                continue;
            }

            $event = $this->em->getRepository(Event::class)->findOneBy([
                'source'     => $sourceEntity,
                'externalId' => $scraped->externalId,
            ]);

            if ($event === null) {
                $event = new Event($sourceEntity, $scraped->externalId, $scraped->name, $scraped->date, $scraped->place);
                $this->em->persist($event);
                $result['created']++;
            } else {
                $result['updated']++;
            }

            $event->update($scraped->name, $scraped->date, $scraped->place, $scraped->region, $scraped->url);
            $event->replaceDistances($scraped->distances);

            // Save after every event, so a crash halfway through keeps what was already scraped.
            $this->em->flush();
        }

        $sourceEntity->markScraped(new DateTimeImmutable());
        $this->em->flush();

        return $result;
    }
}

Errors are handled per page. If one race page times out or contains something unexpected, it's recorded as a failure and the importer moves on to the next page. The importer also saves after every event, so a crash halfway through a run doesn't lose the races already scraped.

The command ties it together. It's a regular Symfony Console command, which laminas-cli runs inside our application:

api/src/Scraper/src/Command/ScrapeCommand.php

<?php

declare(strict_types=1);

namespace Scraper\Command;

use Scraper\EventImporter;
use Scraper\EventSource;
use Symfony\Component\Console\Command\Command;
use Symfony\Component\Console\Input\InputInterface;
use Symfony\Component\Console\Input\InputOption;
use Symfony\Component\Console\Output\OutputInterface;

final class ScrapeCommand extends Command
{
    /**
     * @param list<EventSource> $sources
     */
    public function __construct(
        private readonly EventImporter $importer,
        private readonly array $sources,
    ) {
        parent::__construct('events:scrape');
    }

    protected function configure(): void
    {
        $this->setDescription('Scrape running events from all configured sources.')
            ->addOption('limit', null, InputOption::VALUE_REQUIRED, 'Maximum event pages per source (0 = no limit)', '0')
            ->addOption('source', null, InputOption::VALUE_REQUIRED, 'Only scrape the source with this name');
    }

    protected function execute(InputInterface $input, OutputInterface $output): int
    {
        $limit    = (int) $input->getOption('limit');
        $only     = $input->getOption('source');
        $exitCode = Command::SUCCESS;

        foreach ($this->sources as $source) {
            if ($only !== null && $source->name() !== $only) {
                continue;
            }

            $output->writeln(sprintf('<info>%s</info>', $source->name()));

            try {
                $result = $this->importer->import($source, $limit);
            } catch (\Throwable $e) {
                // Discovery failed (e.g. the sitemap is down); try the next source.
                $output->writeln(sprintf('<error>  failed: %s</error>', $e->getMessage()));
                $exitCode = Command::FAILURE;
                continue;
            }

            $output->writeln(sprintf(
                '  created %d, updated %d, skipped %d, failed %d',
                $result['created'],
                $result['updated'],
                $result['skipped'],
                count($result['failed']),
            ));
            foreach ($result['failed'] as $failure) {
                $output->writeln('  <comment>' . $failure . '</comment>');
            }
        }

        return $exitCode;
    }
}

The --limit option is for development: --limit=10 fetches only ten event pages per source. --source runs a single source.

Two classes need factories because they need configuration. The page fetcher gets the user agent and delay:

api/src/Scraper/src/PageFetcherFactory.php

<?php

declare(strict_types=1);

namespace Scraper;

use Psr\Container\ContainerInterface;
use Symfony\Component\HttpClient\HttpClient;

final class PageFetcherFactory
{
    public function __invoke(ContainerInterface $container): PageFetcher
    {
        $config = $container->get('config')['scraper'];

        $http = HttpClient::create([
            'headers'       => ['User-Agent' => $config['user_agent']],
            'timeout'       => 15,
            'max_redirects' => 3,
        ]);

        return new PageFetcher($http, $config['delay_ms']);
    }
}

And the command gets the list of sources from configuration, so adding a third source means writing one adapter class and adding one line to the config:

api/src/Scraper/src/Command/ScrapeCommandFactory.php

<?php

declare(strict_types=1);

namespace Scraper\Command;

use Psr\Container\ContainerInterface;
use Scraper\EventImporter;

final class ScrapeCommandFactory
{
    public function __invoke(ContainerInterface $container): ScrapeCommand
    {
        $sourceClasses = $container->get('config')['scraper']['sources'];

        return new ScrapeCommand(
            $container->get(EventImporter::class),
            array_map($container->get(...), $sourceClasses),
        );
    }
}

Replace the generated config provider:

api/src/Scraper/src/ConfigProvider.php

<?php

declare(strict_types=1);

namespace Scraper;

use Laminas\ServiceManager\AbstractFactory\ReflectionBasedAbstractFactory;

final class ConfigProvider
{
    public function __invoke(): array
    {
        return [
            'dependencies' => $this->getDependencies(),
            'laminas-cli'  => [
                'commands' => [
                    'events:scrape' => Command\ScrapeCommand::class,
                ],
            ],
            'scraper'      => [
                // Identify yourself: site owners should be able to see who is scraping and how to reach you.
                'user_agent' => 'RunningEventsTutorial/1.0 (+mailto:you@example.com)',
                'delay_ms'   => 1000,
                'sources'    => [
                    Source\TekaskiKoledarSource::class,
                    Source\RekreacijaSource::class,
                ],
            ],
        ];
    }

    public function getDependencies(): array
    {
        return [
            'factories' => [
                PageFetcher::class                 => PageFetcherFactory::class,
                Command\ScrapeCommand::class       => Command\ScrapeCommandFactory::class,
                EventImporter::class               => ReflectionBasedAbstractFactory::class,
                Source\TekaskiKoledarSource::class => ReflectionBasedAbstractFactory::class,
                Source\RekreacijaSource::class     => ReflectionBasedAbstractFactory::class,
            ],
        ];
    }
}

Three things are new compared to Parts 1 and 2:

ReflectionBasedAbstractFactory builds a class by looking at its constructor and fetching each parameter from the container by its type. It replaces a hand-written factory for any class whose constructor takes only services, like the importer and the two adapters.
The laminas-cli section registers our command under the name events:scrape.
The scraper section holds settings. Change the user_agent to include your own contact address before running the scraper against real sites.

Try It

Run a short scrape first, three pages per source:

docker compose run --rm php vendor/bin/laminas events:scrape --limit=3

tekaski-koledar.si
  created 3, updated 0, skipped 0, failed 0
rekreacija.si
  created 2, updated 0, skipped 1, failed 0

The skipped page is the race in Croatia. Run the command again and "created" becomes "updated": the external IDs let the importer recognize races it has already saved.

The sitemap lists races oldest first, so the first few pages are January races. Ask the API for the whole year to see them:

curl "http://localhost:8080/events?from=2026-01-01"

A full run, without --limit, takes several minutes because of the one-second delay between requests. That's intended. When we ran it, it stored 315 races from tekaški-koledar.si and 35 from rekreacija.si, with no failures. The only skipped page was the race in Croatia, and only one race ended up without a region, because its address on the site is just "Slovenija".

Step 7: Tests Without the Network

Scrapers break when websites change, and you want to find out from a failing test rather than from an empty event list. Because the adapters' parse() methods take HTML as a string, we can test them with saved, shortened copies of real pages. The skeleton from Part 1 already includes PHPUnit and a test/AppTest/ folder:

api/test/AppTest/Scraper/ParsersTest.php

<?php

declare(strict_types=1);

namespace AppTest\Scraper;

use PHPUnit\Framework\Attributes\DataProvider;
use PHPUnit\Framework\TestCase;
use Scraper\PageFetcher;
use Scraper\Parser\DistanceParser;
use Scraper\Parser\Region;
use Scraper\Source\RekreacijaSource;
use Scraper\Source\TekaskiKoledarSource;
use Symfony\Component\HttpClient\MockHttpClient;

final class ParsersTest extends TestCase
{
    /** @return iterable<string, array{string, list<int>}> */
    public static function distances(): iterable
    {
        yield 'several km'       => ['6 km · 12 km · 18 km', [18000, 12000, 6000]];
        yield 'decimal comma'    => ['0,2 km · 4 km', [4000, 200]];
        yield 'half marathon'    => ['21,1 km', [21100]];
        yield 'meters'           => ['otroški tek 800 m', [800]];
        yield 'words'            => ['maraton in polmaraton', [42195, 21097]];
        yield 'no distances'     => ['niso dodane', []];
    }

    /** @param list<int> $expected */
    #[DataProvider('distances')]
    public function testDistanceParser(string $text, array $expected): void
    {
        self::assertSame($expected, DistanceParser::parse($text));
    }

    public function testRegionNormalization(): void
    {
        self::assertSame('Gorenjska', Region::normalize('Gorenjska'));
        self::assertSame('Jugovzhodna Slovenija', Region::normalize('Jugovzhodna'));
        self::assertNull(Region::normalize('Primorsko-goranska'));
        self::assertNull(Region::normalize(null));
    }

    public function testTekaskiKoledarPage(): void
    {
        $html = <<<'HTML'
            <html><head>
            <script type="application/ld+json">
            {"@context":"https://schema.org","@type":"Event","name":"Zimski trail tek Bohinj",
             "startDate":"2026-03-07","location":{"@type":"Place","name":"Bohinj",
             "address":"Bohinj, Gorenjska, Slovenija"},
             "description":"Zimski trail tek Bohinj – 7. marec 2026, Bohinj. Razdalje: 6 km · 12 km · 18 km. Podrobnosti ..."}
            </script>
            </head><body></body></html>
            HTML;

        $url   = 'https://tekaski-koledar.si/tek/2026/r000123-zimski-trail-tek-bohinj/';
        $event = (new TekaskiKoledarSource($this->fetcher()))->parse($url, $html);

        self::assertNotNull($event);
        self::assertSame('2026/r000123', $event->externalId);
        self::assertSame('Zimski trail tek Bohinj', $event->name);
        self::assertSame('2026-03-07', $event->date->format('Y-m-d'));
        self::assertSame('Bohinj', $event->place);
        self::assertSame('Gorenjska', $event->region);
        self::assertSame([18000, 12000, 6000], $event->distances);
    }

    public function testTekaskiKoledarFindsRegionInLongerAddress(): void
    {
        $html = <<<'HTML'
            <script type="application/ld+json">
            {"@type":"Event","name":"Tek na Veliko planino","startDate":"2026-07-12",
             "location":{"@type":"Place","name":"Kamnik",
             "address":"Kamnik, Velika planina, Gorenjska, Slovenija"},
             "description":"Razdalje: niso dodane."}
            </script>
            HTML;

        $event = (new TekaskiKoledarSource($this->fetcher()))
            ->parse('https://tekaski-koledar.si/tek/2026/r000200-tek-na-veliko-planino/', $html);

        self::assertNotNull($event);
        self::assertSame('Gorenjska', $event->region);
        self::assertSame([], $event->distances);
    }

    public function testRekreacijaPage(): void
    {
        $html = <<<'HTML'
            <html><head>
            <script type="application/ld+json">
            {"@context":"https://schema.org","@type":"SportsEvent","name":"16. Dobrepoljski tek",
             "startDate":"2026-10-10T07:30:00+00:00","location":{"@type":"Place","name":"Dobrepolje",
             "address":{"@type":"PostalAddress","addressLocality":"Dobrepolje",
             "addressRegion":"Osrednjeslovenska","addressCountry":"SI"}}}
            </script>
            </head><body></body></html>
            HTML;

        $url   = 'https://www.rekreacija.si/koledar/16-dobrepoljski-tek/';
        $event = (new RekreacijaSource($this->fetcher()))->parse($url, $html);

        self::assertNotNull($event);
        self::assertSame('16-dobrepoljski-tek', $event->externalId);
        self::assertSame('2026-10-10', $event->date->format('Y-m-d'));
        self::assertSame('Osrednjeslovenska', $event->region);
        self::assertSame([], $event->distances);
    }

    public function testRekreacijaFindsRegionInLocality(): void
    {
        $html = <<<'HTML'
            <script type="application/ld+json">
            {"@type":"SportsEvent","name":"Novomeški polmaraton","startDate":"2026-11-28T10:00:00+00:00",
             "location":{"@type":"Place","name":"Novo mesto",
             "address":{"@type":"PostalAddress","addressLocality":"Novo mesto, Jugovzhodna","addressCountry":"SI"}}}
            </script>
            HTML;

        $event = (new RekreacijaSource($this->fetcher()))
            ->parse('https://www.rekreacija.si/koledar/novomeski-polmaraton/', $html);

        self::assertNotNull($event);
        self::assertSame('Novo mesto', $event->place);
        self::assertSame('Jugovzhodna Slovenija', $event->region);
    }

    public function testRekreacijaSkipsEventsOutsideSlovenia(): void
    {
        // Baška is in Croatia, but the page says "SI". There is no Slovenian region, so we skip it.
        $html = <<<'HTML'
            <script type="application/ld+json">
            {"@type":"SportsEvent","name":"Baška Outdoor Festival","startDate":"2026-10-09",
             "location":{"@type":"Place","name":"Baška-Krk",
             "address":{"@type":"PostalAddress","addressLocality":"Baška-Krk","addressCountry":"SI"}}}
            </script>
            HTML;

        $event = (new RekreacijaSource($this->fetcher()))
            ->parse('https://www.rekreacija.si/koledar/baska/', $html);

        self::assertNull($event);
    }

    private function fetcher(): PageFetcher
    {
        // parse() never makes HTTP requests; this client would fail the test if it did.
        return new PageFetcher(new MockHttpClient([]), 0);
    }
}

docker compose run --rm php vendor/bin/phpunit --testdox test/AppTest/Scraper

Parsers (AppTest\Scraper\Parsers)
 ✔ Distance parser with data set "several km"
 ✔ Distance parser with data set "decimal comma"
 ✔ Distance parser with data set "half marathon"
 ✔ Distance parser with data set "meters"
 ✔ Distance parser with data set "words"
 ✔ Distance parser with data set "no distances"
 ✔ Region normalization
 ✔ Tekaski koledar page
 ✔ Tekaski koledar finds region in longer address
 ✔ Rekreacija page
 ✔ Rekreacija finds region in locality
 ✔ Rekreacija skips events outside slovenia

OK (12 tests, 29 assertions)

When a site changes its markup, save a fresh copy of one page, add it as a test case, and fix the adapter until the test passes.

Step 8: Run It Every Day

The scraper gets its own container, built from the same PHP image. It runs the command, sleeps for 24 hours (86,400 seconds), and repeats. Replace docker-compose.yml:

docker-compose.yml

services:
  php:
    build: ./docker/php
    volumes:
      - ./api:/var/www/api
    environment:
      DB_HOST: mysql
      DB_NAME: running_events
      DB_USER: app
      DB_PASSWORD: app
    depends_on:
      mysql:
        condition: service_healthy

  worker:
    build: ./docker/php
    volumes:
      - ./api:/var/www/api
    environment:
      DB_HOST: mysql
      DB_NAME: running_events
      DB_USER: app
      DB_PASSWORD: app
    command: ["sh", "-c", "while true; do vendor/bin/laminas events:scrape; sleep 86400; done"]
    restart: unless-stopped
    depends_on:
      mysql:
        condition: service_healthy

  nginx:
    image: nginx:stable-alpine
    ports:
      - "8080:80"
    volumes:
      - ./api:/var/www/api:ro
      - ./docker/nginx/default.conf:/etc/nginx/conf.d/default.conf:ro
    depends_on:
      - php

  mysql:
    image: mysql:8.4
    environment:
      MYSQL_DATABASE: running_events
      MYSQL_USER: app
      MYSQL_PASSWORD: app
      MYSQL_ROOT_PASSWORD: root
    ports:
      - "3307:3306"
    volumes:
      - mysql-data:/var/lib/mysql
    healthcheck:
      test: ["CMD", "mysqladmin", "ping", "-h", "127.0.0.1", "-uapp", "-papp"]
      interval: 5s
      timeout: 3s
      retries: 20

volumes:
  mysql-data:

docker compose up -d
docker compose logs -f worker

The worker starts a full scrape immediately, then waits a day. restart: unless-stopped brings it back if it crashes or the machine reboots. Because each scrape is a new PHP process, nothing from a previous run, such as Doctrine's in-memory state, carries over to the next.

A sleep loop is the simplest scheduler there is. Its one drawback is that the schedule shifts: the next run happens 24 hours after the container started, not at a fixed time of day. If you need "every day at 3:00", use cron or a scheduler like Supercronic in the container instead.

Common Pitfalls

"Skipped" can hide lost data. Both adapter bugs described above produced no errors: events were simply saved without a region, or skipped. We found them only by comparing what each source lists with what ended up in the database, and by looking at every skipped page. Do that once for each new source, and again whenever the counts change unexpectedly.

Selectors break silently. A redesign that renames rekol-row leaves the rekreacija.si adapter finding zero events, with no error. Watch the counts in the output: a source that suddenly reports "created 0, updated 0" needs attention. Structured data changes far less often than CSS classes, which is why both adapters rely on it for the event details.

Failures nobody sees. The worker's output goes to the container log. Check it with docker compose logs worker, or forward it to whatever log system you use. The command also exits with a failure code when a whole source fails, which a proper scheduler can alert on.

HTML entities in names. Text inside structured data can contain entities like – instead of "–". The rekreacija.si adapter decodes them with html_entity_decode(); check names from any new source for the same problem.

Database errors stop the run. If saving fails, for example because a name is longer than the 255-character column, Doctrine closes its entity manager and the rest of that run fails. The next day's run starts fresh. If it happens repeatedly, add the offending page to the tests and make the adapter shorten or skip it.

Scraping too fast. Lowering delay_ms to speed up development is tempting. Use --limit instead; the delay protects the sites you depend on.

What You Have Now

a scraper with two adapters: one using a sitemap, one using paginated listing pages, both reading schema.org structured data
shared parsers for distances, regions and structured data, covered by tests that don't touch the network
an importer that inserts or updates races, records per-page failures, and remembers when each source was last scraped
a worker container that refreshes the data every 24 hours

api/
├── src/Scraper/src/
│   ├── Command/
│   │   ├── ScrapeCommand.php
│   │   └── ScrapeCommandFactory.php
│   ├── ConfigProvider.php
│   ├── EventImporter.php
│   ├── EventSource.php
│   ├── PageFetcher.php
│   ├── PageFetcherFactory.php
│   ├── Parser/
│   │   ├── DistanceParser.php
│   │   ├── JsonLd.php
│   │   └── Region.php
│   ├── ScrapedEvent.php
│   └── Source/
│       ├── RekreacijaSource.php
│       └── TekaskiKoledarSource.php
└── test/AppTest/Scraper/ParsersTest.php

Next, in Part 4: the API now holds real data, and a frontend is about to depend on it. We'll describe the API in an OpenAPI specification, serve interactive documentation, and make the API reject requests that don't match the contract.