Loading Gists in a NextJS Application

Introduction

All my blog posts are available on my personal website. Under the hood, they are saved as markdown files.

It allows me to easily write the posts, but when I’m cross-posting on other platforms, I don’t have the luxury to get the syntax highlighting and the right colors for the code snippets.

To make it prettier on every platform, I’m using Github Gists. It renders well everywhere, but I don’t want to have Github’s code blocs styles on my blog but I prefer something that match my website design. To do so, I have to adapt the markdown parser I use to load and render the code from gists in the way I prefer.

In this post, we will see how to create a blog post using markdown files, how to create and integrate a remark plugin, highlight the recovered code it with react-syntax-highlighter and keep the performances as good as possible.

Initial Situation: Loading Markdown in a NextJS Page

Before getting into the code for loading gists, we first need to make the blog to load markdown files. To do so, it needs two different items:

• A markdown file containing the blog post
• A NextJS page containing the markdown loading, parsing and rendering

Let’s start with the markdown file

What Does The Markdown File Looks Like

The markdown file is separated in two different parts: the metadata and the content itself.

The metadata are here to give informations on the author, the date, the language used (I have posts both in french and english), the URL path, tags, etc.

These informations are isolated from the content by three dashes — at the begin and three and the end.

Here is a light sample of what a blog post can look like:

---
creator: Bruno Sabot
date: 2020-20-20
lang: en
path: /posts/2020/a-sample-blog-post
title: A sample blog post
---

# Introduction

Here is the introduction

# Content

Here is the content

# Conclusion

Here is the conclusion

The file is very simple. If you are not familiar with markdown, you can read this doc from Github to get started: you will learn how to create lists, links, add image and more.

In the reality of my bog, I have more metadata fields to handle canonical links, image, tags and more. I removed them from that post as it is unnecessary noise for its topic.

Loading The Markdown File

The first part to render a markdown file is to load it. We are using basic NodeJS code to:

• Lookup all files in a folder
• Filter the md/mdx files only
• Load the file

It might be overkill, but on my blog, I need to get every posts to find the most similar posts of my current article, based on a calculation on every common tags they have. I will have to load every post to find out anyway.

Here is the code:

import fs from "fs";
import path from "path";

const POSTS_PATH = path.join(process.cwd(), "posts");

const posts = fs
  .readdirSync(POSTS_PATH)
  .filter((path) => /\.mdx?$/.test(path))
  .map((path) => fs.readFileSync(POSTS_PATH + "/" + path));

If you are creating a simpler blog, you might prefer to directly load the right file instead of all of them. To do so, you just need to use the fs.readFileSync function to get your file.

Now the posts are loaded, we need to search inside the file which one is the right one for our current page.

To do so, I’m looking for the post with same path as requested, but I first need to parse my markdown for metadata. I’m using a package called grey-matter to get them.

Here is the code:

import fs from "fs";
import path from "path";

const POSTS_PATH = path.join(process.cwd(), "posts");

function loadGreyMatterPost(params) {
  if (
    params === undefined ||
    params.year === undefined ||
    params.slug === undefined
  ) {
    throw new Error("Not Found");
  }

  const posts = fs
    .readdirSync(POSTS_PATH)
    .filter((path) => /\.mdx?$/.test(path))
    .map((path) => fs.readFileSync(POSTS_PATH + "/" + path))
    .map((source) => matter(source));

  const post = posts.find(
    (post) => post.data.path === `/posts/${params.year}/${params.slug}`
  );

  if (post === undefined) {
    throw new Error("Not Found");
  }

  return [post, posts];
}

The loadGreyMatterPost is a function returning a file representing our markdown at the grey-matter format. We still have to transform it into an HTML code to use it. As explained earlier, I am also returning all the posts to make matches based on the tags. It will however not explained in this post.

Since I’m using NextJS, I can now use the package next-mdx-remote which can make the transformation from the grey-matter format to a MDX one.

Here is the code:

import { serialize } from "next-mdx-remote/serialize";

async function transformGreyMatterToMDX(post) {
  const mdxSource = await serialize(post.content, {
    mdxOptions: {
      remarkPlugins: [],
      rehypePlugins: [],
    },
    scope: post.data,
  });

  return mdxSource;
}

Adding The Loaded Data Inside a NextJS Route

To avoid enormous and slow payload, the loading part is added inside the NextJS route’s getStaticProps method. It will only parse the files at the compile time, which is OK with markdown, and serve static pages the faster possible.

Here is the code:

import { GetStaticProps } from "next";
import loadGreyMatterPost from "../lib/loadGreyMatterPost";
import transformGreyMatterToMDX from "../lib/transformGreyMatterToMDX";

export const getStaticProps: GetStaticProps = async ({ params }) => {
  try {
    const [post] = loadGreyMatterPost(params);
    const mdxSource = transformGreyMatterToMDX(postt);

    return {
      props: {
        source: mdxSource,
        post: {
          creator: post.data.creator,
          date: post.data.date.toString(),
          lang: post.data.lang,
          path: post.data.path,
          title: post.data.title,
        },
      },
    };
  } catch (e) {
    return { notFound: true };
  }
};

Display the blog post in the page

Now we have all the required informations, we can include them in the NextJS page.

It will need two external components:

• MDXProvider which is a component that can handle defaults informations for every children markdown document
• MDXRemote which transform the MDX content to HTML React components

Here is the code:

import React from "react";
import { MDXProvider } from "@mdx-js/react";
import { MDXRemoteSerializeResult } from "next-mdx-remote";
import { MDXRemote } from "next-mdx-remote";

/* getStaticProps method here */

interface IBlogPost {
  creator: string;
  date: string;
  lang: string;
  path: string;
  title: string;
}

interface IBlogProps {
  post: IBlogPost;
  source: MDXRemoteSerializeResult<Record<string, unknown>>;
}

const components = {};

const Blog: React.FC<IBlogProps> = ({ source, post }) => {
  return (
    <MDXProvider components={components}>
      <article>
        <h1>{post.title}</h1>
        <h2>
          By {post.creator} on {post.date}
        </h2>
        <MDXRemote {...source} />
      </article>
    </MDXProvider>
  );
};

export default Blog;

And it is done! We are now able to write markdown blog post and get them rendered as HTML in our NextJS application!

Creating a Remark plugin

The AST transformation

A remark plugin is a function that return another function. That last one gets an AST -Abstract Syntax Tree- representing the markdown document.

The AST can get numerous fields, but here is a simple typescript interface of how it is structured:

interface IPosition {
  line: number;
  column: number;
  offset: number;
}

interface IAST {
  children?: IAST[];
  depth?: number;
  meta?: string | null;
  lang?: string;
  type: string;
  value?: string;
  name?: string;
  attributes?: IAST[];
  position?: {
    start: IPosition;
    end: IPosition;
  };
}

When using markdown, the AST for code does not have children or attributes, while the paragraph AST does not have any value. To understand how it is built, imagine the following markdown:

# Introduction

Here is my content

```
// Here is a code bloc
```

The associated AST will be:

{
  "type": "root",
  "children": [
    {
      "type": "heading",
      "depth": 1,
      "children": [
        {
          "type": "text",
          "value": "Introduction",
          "position": {
            "start": { "line": 2, "column": 3, "offset": 3 },
            "end": { "line": 2, "column": 15, "offset": 15 }
          }
        }
      ],
      "position": {
        "start": { "line": 2, "column": 1, "offset": 1 },
        "end": { "line": 2, "column": 15, "offset": 15 }
      }
    },
    {
      "type": "paragraph",
      "children": [
        {
          "type": "text",
          "value": "Here is my content",
          "position": {
            "start": { "line": 4, "column": 1, "offset": 17 },
            "end": { "line": 4, "column": 19, "offset": 35 }
          }
        }
      ],
      "position": {
        "start": { "line": 4, "column": 1, "offset": 17 },
        "end": { "line": 4, "column": 19, "offset": 35 }
      }
    },
    {
      "type": "code",
      "lang": "javascript",
      "meta": null,
      "value": "Here is a code bloc",
      "position": {
        "start": { "line": 6, "column": 1, "offset": 37 },
        "end": { "line": 8, "column": 4, "offset": 74 }
      }
    }
  ],
  "position": {
    "start": { "line": 1, "column": 1, "offset": 0 },
    "end": { "line": 9, "column": 1, "offset": 75 }
  }
}

With this in mind, we can think about how we are going to include our gist.

Fetching a Gist From Github

I chose to include the gist in an inline block code, with a prefix gist: and then the username and ID. It will look like this:

gist:brunosabot/00000000000000000000000000000000

When rendering this content, the generated AST will be composed of a paragraph node with an inlineCode child. What we need to do is to replace the paragraph the loaded code from Github.

As we need code from Github, we will also need to fetch the gist. A gist can be made of one to several files: we will first need to iterate on the file list then load each one of them.

Here is the code:

import fetch from "node-fetch";

interface GistJsonResponse {
  files: string[];
}

async function loadGist(id: string) {
  const url = `https://gist.github.com/${id}`;
  const jsonResponse = await fetch(`${url}.json`);
  const jsonData = await (jsonResponse.json() as Promise<GistJsonResponse>);

  const data = await Promise.all(
    jsonData.files.map((file) =>
      fetch(`${url}/raw/?file=${file}`).then((r) => r.text())
    )
  );

  return [data, jsonData];
}

The first fetch query gets a JSON containing metadata for the gist, especially the file list in the gist.

When the list is recovered, we iterate them to load as text the code associated to every file.

Visiting And Updating The AST

The gist plugin should alter the content of the original AST. We are going to visit the document nodes and update it as we want.

To visit the different nodes, we use the async-unist-util-visit package. it will apply a method on every AST child that match the requested type.

As explain before, we are going to look for inlineCode, starting with gist:, inside a paragraph node, then parse it and load the gist.

But let’s focus on the visiting code:

import visit from "async-unist-util-visit";

const gist = () => async (ast: IAST) => {
  const promises: Promise<void>[] = [];

  const createGist = (node: IAST) => {
    if (node.children === undefined) return;
    if (node.children.length === 0) return;

    const hasGist = node.children.some(
      ({ type, value }) =>
        type === "inlineCode" && value?.startsWith("gist:") && value?.length > 5
    );

    if (hasGist) {
      const promise = loadAndTransformGist(node, node.children[0]).then(
        (newNode) => {
          Object.assign(node, newNode);

          if (node.children?.length === 0) {
            delete node.children;
          }
        }
      );

      promises.push(promise);
    }
  };

  await visit(ast, "paragraph", createGist);
  await Promise.all(promises);

  return;
};

One important thing to notice is that the AST must be mutated and not returned. Since visit is a promise but does not wait for the callback to resolve, we need to use a hack consisting on adding the work in a promise list and wait them to resolve before the plugin method to end.

In the code snippet:

• The node.children.some method is looking for a gist snippet in the code
• The Object.assign method is the way to update the content of the object without loosing the reference
• The delete is not mandatory, but I like to keep a clear object with only the required fields.

In the plugin, we are using a loadAndTransformGist method that is called when we detect a gist snippet in the code.

As it written in the function name, this method will load the gist and make the proper transformation to be included in the AST.

Here its code:

async function loadAndTransformGist(parent: IAST, item: IAST): Promise<IAST> {
  if (item.value === undefined) return item;

  const gist = item.value.substring(5).trim();
  const [data, jsonData] = await loadGist(gist);

  if (data.length === 1) {
    return getGistAST(jsonData.files[0], data[0]);
  }

  return {
    type: parent.type,
    children: data.map((file, index) =>
      getGistAST(jsonData.files[index], file)
    ),
  };
}

In this snippet, the first part is to check it the snippet is valid, which is basically checking if the AST value is empty or not.

I could make few more checks, including one to validate that the gist value is correct. I choose not to since I’m often checking the rendering of my post: I can see very quick when the ID is wrong since the app will either crash or don’t show anything.

I’m then extracting the gist ID, load it with the previously created loadGist method.

Then, I have two possibilities:

First, there is only one file in the gist. I will just replace the current paragraph with a bock sample. The getGistAST will give me the code AST that I will return for the calling method.

Second, there is multiple files in the gist. I will iterate on them and add them as children of the paragraph node.

The getGistAST is basically a mapping method with the following code:

function getGistAST(file: string, value: string): IAST {
  return {
    type: "mdxJsxFlowElement",
    name: "Gist",
    attributes: [
      { type: "mdxJsxAttribute", name: "file", value: file },
      { type: "mdxJsxAttribute", name: "code", value: value.trim() },
      {
        type: "mdxJsxAttribute",
        name: "lang",
        value: getLanguage(file.split(".").at(-1)),
      },
    ],
    children: [],
  };
}

Translated to english, this AST node is a <Gist> component, with:

• A file attribute representing the file name
• A code attribute which is the actual code
• A lang attribute containing the file language, calculated from a mapping method

And here is the mapping method:

const MAP_LANGUAGE = {
  yml: "yaml",
  Dockerfile: "docker",
  eslintrc: "json",
  js: "javascript",
  ts: "typescript",
  stylelintrc: "json",
  prettierrc: "json",
};

function getLanguage(inputLanguage: string | undefined) {
  if (inputLanguage === undefined) return "text";

  if (inputLanguage in MAP_LANGUAGE) {
    return MAP_LANGUAGE[inputLanguage as keyof typeof MAP_LANGUAGE];
  }

  return inputLanguage;
}

And everything is done, we just need to update our transformGreyMatterToMDX method, adding the plugin:

import { serialize } from "next-mdx-remote/serialize";
import gist from "./lib/gist";

async function transformGreyMatterToMDX(post) {
  const mdxSource = await serialize(post.content, {
    mdxOptions: {
      remarkPlugins: [gist],
      rehypePlugins: [],
    },
    scope: post.data,
  });

  return mdxSource;
}

Using a Custom Rendering Component

Now the markdown is able to read gist snippets, there is a last step to make the actual code rendering: we need to create a Gist component to make the proper display.

To do so, I will use the React Syntax Highlighter package.