First Sonic Pi track, made with Literate Programming

Hi there!

Here is my first track made with Sonic Pi, and also my first real piece of music at all.

House Experiment 001 - Sonic Pi

The track was written using Literate Programming. This means there is a textual description of the project, including notes/scores, which contains the complete code of the track:

https://mlange-42.github.io/Sonic-LP/first-steps/house-bounce.html

Using a command line tool, the code is extracted from the documentation into a code file usable with Sonic Pi.

Please bear with me if I used quite some nonsense terminology in the descriptions, or made mistakes in the scores. I have literaly no idea about musical theory, except that I know how notes of different length look like. :wink: So, I would be happy about any hints on false terms used there.

The extracted code is available on GitHub:

EDIT: Is it possible to embed an external track from TheFuture.fm here?

7 Likes

OMG this is such a cool project! I didn’t realise tools like these existed. I love how you’re presenting your ideas, and I hope to hear more from you.

Your descriptions look fine at first glance :+1:

1 Like

Many thanks, @d0lfyn !

Literate Programming has a relatively long tradition, but certainly in a narrow niche. E.g., TeX was written this way, starting in 1981.

Due to the fact that I found no such tool that is easy to use and uses a modern and simple syntax (here: Markdown), I startet Yarner one year ago.

2 Likes

I’m certainly interested to see what becomes of Yarner and this approach to programming!

Great stuff! Yes, literal programming was invented by Donald Knuth, because he wanted code and documentation of the code to be the same thing. But TeX was so extremely well done that it became a tool for creating high quality documents on its own, without the code. Most scientific publications are written in TeX (or LaTeX) nowadays.

A similar aproach was taken with JuPyter notebooks, a programming and documentation format that tightly integrates code and documentation (markdown, HTML and LaTex). JuPyter notebooks already integrate with the runtime environment, so that you can run the code directly from the intercative notebook without actually extracting the code.

Your solution Yarner is more in the spirit of TeX with marking up the code parts and then extract those. Again, super interesting stuff, congrats! And quite a complex piece of music as a starter :+1:

3 Likes

Hi,

Thanks for sharing your tool with the sonic pi community

Could you explain the command line to create the output ? How to install the yarner engine ?
Could be interesting for educationnal purpose but even if i have a look at the .md source, i don’t see how these file are structured. i guess we have to respect some nomenclature, some yarner keywords right ?

Cheers

Thanks, @Nechoj and @nlb for your feedback!

Literate Programming in general

@Nechoj Indeed, I forgot to mention the probably most widespread LP systems, which are Jupyter notebooks and RMarkdown. Both can run the code directly and allow to include the results (e.g. plots) into the document. However, both are more targeted on scripting, like for data analysis or other statistics.

With Yarner, I was more looking into the direction of writing software applications in compiled languages. Particularly simulation models in most of my personal use cases.

Of course, the approach taken makes things much easier as I do not need to care at all about the programming languages used, nor how to compile or run the code. This needs to be done by the user with the extracted code, just as usual.

Yarner usage

@nlb There is a comprehensive and detailed user guide that explains everything. But I can try to give a brief overview here.

Installation

If you have Rust installed, simply run

cargo install yarner

Otherwise, download it for your platform from the releases, extract it somewhere, and ideally put it on the PATH.

For more deteil, see chapter Installation of the guide.

Command line usage

Command line usage is extremely simple. To set up a new project, navigate into your to-be project folder and simply run

yarner init

You then get a small but working example project which you can start with. To extract the code and create the “clean” documentation, simply run this, also inside the project’s folder:

yarner

And that’s it… with the default configuration, you then find the extracted code in sub-folder code, and the clean docs in sub-folder docs (with the simple example project, “clean” docs look exactly the same as the Markdown sources).

As said, the code output can be compiled or run as usual. Further, the user is free to convert the Markdown docs to HTML or PDF, or simply push them to a Git forge where Markdown is rendered by default.

Markdown syntax

Yarner tries to be completely compatible with standard Markdown. The basic functionality is based on named code blocks and macros.

Code blocks are named by their first line of code, prefixed by a user-configurable sequence. For the project presented here, I used #- as a prefix, as it renders as a comment with Ruby syntax highlighting. As an example, a code block named Some block would be written like this:

```ruby
#- Some block
Your normal code in Some block...
```

Then, named code blocks can be used with macros (syntax is also configurable), which results in the macro being replaced by the content of the referenced block in the code output. As an example:

```ruby
#- Main block
Some code here...
# ==> Some block.
```

The last line is a macro call to Some block. As a result, the content of Some block is put where the macro is. Thus, the extracted code would look like this:

Some code here...
Your normal code in Some block...

Note: Only fenced code blocks (surrounded with ``` or ~~~) are supported, but not the indentation syntax (indent by four spaces, instead of fencing)

This allows you to structure the code for human readers, and Yarner takes care to re-structure it for the machine.

There are a lot more features, e.g. to allow to work with multiple source files, or to generate multiple code output files, but this is the fundamental concept how is works. For more details, see the user guide.

2 Likes

Cool ideas indeed! I must say though, that for me personally I think I need to really just code a track completely. I used trackers before (Amiga, PC etc) and I noticed that seeing the patterns restrict my creative process, a track evolves while listening to combinations of tones and effects for me.
But that is just personal, I really like this project of yours for the sake of it being a very good idea!

1 Like

Of course, I also started experimenting in Sonic Pi directly, and only after all the basics were set moved it to Literate Programming.

…if that is what you mean with:

I think I need to really just code a track completely

Ah, ok… this is probably wrong and you mean the string patterns for drums and track structure…

EDIT: Actually, I just discovered that the uploaded track was not replaced by the latest version properly. For those who listened to it instead of pasting the code into SP, the best variation in the bass line (e.g. at 5:50) was missing until a few minutes ago…

I mean, if I see the structure of a song, my brain says: no, i dont want that, i will stop being creative now :smiley:
That is why I was so amazed by Sonic PI, as that works for me! I think that is because I see music differently maybe and that’s why sonic pi suits me in that aspect

Hello and thank you for your very cool composition
and good documentaion of this too.

I am trying to ‘yarner’ your seemingly incomplete ‘book’
but this seems not to work just with the yarner 0.6
(Windows program; .exe).

C:\Users\ab2\Documents\Sonic Pi\Sonic-LP-main>yarner
[INFO ] Extracting code from first-steps.md
[WARN ]   No entrypoint for file first-steps.md, skipping code output.
[INFO ] Extracting code from first-steps/house-bounce.md
[INFO ]   Skipping unchanged file ../code/House/HouseBounce.rb
[INFO ] Extracting code from introduction.md
[WARN ]   No entrypoint for file introduction.md, skipping code output.
[INFO ] Extracting code from SUMMARY.md
[WARN ]   No entrypoint for file SUMMARY.md, skipping code output.
[INFO ] Running plugin 'block-links'
[ERROR] Failed to run plugin command 'yarner-block-links': Das System kann die angegebene Datei nicht finden. (os error 2)

Probably I would need to install more of ‘yarner’?
Thank you for some advise.

@Buce You need to install the plugin yarner-block-links. See GitHub - mlange-42/yarner-block-links: Yarner plugin that creates links to all referenced end referencing blocks

Thank you very much for the quick answer.

This works fine now.

How did you render/generate (the book)
output then?

Somehow you say that github is rendering it automatically but could
I extract this locally to have some html-files arround?

Thank for comments.

PS: You seem to have to unfinished chapters (dubstep-001 and morricone-001)…

You first run Yarner, then you run mdBook (runs on Yarner’s output).

Should be as simple as this:

yarner
mdbook build

For more details, see the CI script: Sonic-LP/publish.yml at main · mlange-42/Sonic-LP · GitHub

You can also see the required plugins there for both, Yarner (yarner-block-links) and mdBook (mdbook-toc).

You can also download all these things manually that are retieved using curl there.

… and if you want to have it built and published by/on GitHub, just use the CI script. You need to enable GitHub Pages for it to work.

Unfortunately is mdbook-toc(v 0.9, 0.8) rejected by a windows virus detection? Any recommendations?

No, never encontered that. Maybe add an exception?