factorio-recipe-analyzer/README.org

Factorio Recipe Analyzer

• Intro
• On this file.
  • Dependencies
• License
  • TODO: include GPLv3 in the repo
  • Credits:
• Lua scripts
  • Export all recipes and ingredients that match whitelist
  • Export all recipes (including products and ingredients)into an org-mode file, categorized by theis recipe subgroup.
• Approach 1: passing through org-mode.
  • Idea:
  • Setup
  • Open csv file
  • Open (create if needed) output csv
  • Make a list of products
    • For each product, create a templated section
      • Parse the csv file:
      • Parse the output.org file, filling it out recursively
• Approach 2: working directly in the csv files
  • Idea/ outline
  • Testing 1:
    • Get csv from Factorio
    • Example outputs that we could use:
      • Sorted by recipes:
      • Sorted by products:
    • Move csv to the right directory
    • Sort for easier human readability
      • Sort the csv file by recipe subgroups
      • Sort the csv file by product name
    • Parse the intermetiate.csv file, filling it out recursively
    • Define raw ingredients by their category
      • We have the following categories we'll be assigning:
      • Temp code:
• New Idea to be given thought
  • Elaboration:
• Approach 3: pulling from Factorio data dump json
  • Get files into working area
  • Testing: Perl script parsing JSON into
    • Init:
    • Run the script:

Intro

This is, ideally, a script which:

1. Generates an analysis of each recipe into its component parts, and delivers insight into the balance and progression of a mod.
2. Allows for easy editing of item, recipe, building, technology properties.
3. Allows those edits to be easily exported into lua files, creating compatibility between various mods. Or at least assisting in making that compatibility happen.

Of course, that's a long shot.

On this file.

This is a literate script. The source code is embedded in these code blocks, and tangled into the script using org-babel. This allows me to write a description of what I want to do, and comment on it without resorting to // /* */ ugly comments.

If you open this org file raw (e.g. by clicking here), you will see that there are a number of different blocks.

There is a rough overview of the structure:

#+name: Name of the following source block.
#+begin_src lang :tangle file.name
[code goes here]
#+end_src

:tangle defines the file into which the code block will be passed to org-babel-tangle. To disable, pass no.

In the begging of the file, we can set global (in the scope of the file) properties, such as the default target for :tangle:

#+PROPERTY: header-args :tangle no

The easiest way to generate outputs from here is to open this file in Emacs, and run M-x org-babel-tangle. If you're using Doom Emacs, the default key binding is C-c C-v C-t.

Important note: the files in ./data are working files. They're not intrinsic parts of the script. Everything you need should ideally be contained within this one single .org file.

However, inevitably, since we're dealing with external data, there are some…

Dependencies

1. Hard dependenciens:

   1. Factorio. Obviously. You need version at least 1.1.87, since that added the --dump-data argument, which we use.
   2. Perl > 5.14, for JSON::PP and Data::Dumper.

2. Soft (useful) dependencies:

   1. Emacs, org-mode, org-babel. These make the workflow much easier, however you don't need these; they only save time.
   2. sh so you can execute the following from within Emacs.
   3. GNU coreutils (cp, mv). These aren't hard depends, since you can move the files manually. This is so you can just C-c C-c over the code blocks.

License

I don't own the source csv files generated by Factorio, nor the mods the script is pulling from. This script itself is GPLv3, with the exception of third-party libraries licensed otherwise.

TODO: include GPLv3 in the repo

Credits:

The following are mods from which .csv files have been generated. The recipe .csv files are included in the data directory.

• Bobingabout's mods: https://github.com/modded-factorio/bobsmods
• Angel's mods: https://github.com/Arch666Angel/mods

Lua scripts

The .csv files used as input are generated in Factorio, by loading a new game with only base and your chosen mod enabled, and running one of the following Lua scripts from the console:

Export all recipes and ingredients that match whitelist

/c
local whitelist = {}
for _, k in pairs({
    "iron-plate",
    "copper-plate",
    "steel-plate",
    "copper-wire",
    "iron-gear-wheel",
    "iron-stick",
    "pipe",
}) do whitelist[k] = true end

local parts = {}
for name, recipe in pairs(game.recipe_prototypes) do
  local history = script.get_prototype_history("recipe", name)
  if history.created == "base" then
    local add = false
    local ingredients = {}
    for _, ingredient in pairs(recipe.ingredients) do
      if whitelist[ingredient.name] then add = true end
      ingredients[#ingredients+1] = ingredient.amount .. "," .. ingredient.name
    end
    if add then
      local item = game.item_prototypes[name] or game.fluid_prototypes[name]
      parts[#parts+1] = "," .. name .. "," .. item.subgroup.name .. "," ..  table.concat(ingredients, ",")
    end
  end
end
game.write_file("recipes.csv", table.concat(parts, "\n"), false)

Export all recipes (including products and ingredients)into an org-mode file, categorized by theis recipe subgroup.

This script outputs ALL available recipes in an org-readable format, for easy overview, sorting, and insight. Note the entire tree is included for each recipe.

/c
local recipes = {}
for name, recipe in pairs(game.recipe_prototypes) do
    local ingredients = {}
    for _, ingredient in pairs(recipe.ingredients) do
        ingredients[#ingredients+1] = ingredient.name .. "," .. ingredient.amount
    end
    local products = {}
    for _, product in pairs(recipe.products) do
        local amount = product.amount or product.amount_min .. "-" .. product.amount_max
        products[#products+1] = product.name .. "," .. amount
    end
    recipes[#recipes+1] = "\n* " .. recipe.subgroup.name .. "\n** " .. name .. "\n*** products" .. "\n- " .. table.concat(products, "\n- ") .. "\n*** ingredients" .. "\n- " ..  table.concat(ingredients, "\n- ")
end
game.write_file("recipes.org", table.concat(recipes, "\n"), false)

The output of this is an org-mode file in the following pattern:

* recipe.subgroup
** recipe.name
*** products
- product1
- product2
- ...
*** ingredients
- ingredient1
- ingredient2
- ...

**

Approach 1: passing through org-mode.

Idea:

Create an org-mode file with all the calculations included, and complete, for a clear overview of recipe progression, subgroups, item inputs and outputs, and total cost.

Setup

Open csv file

Open (create if needed) output csv

For this, we'll likely want to include some metadata, like creation date, mod name, number of ingredients, maybe the total amount of raw mats needed to make one of everything?

Make a list of products

That's the first column in the .csv file. Read first column of the csv file and insert it into the .org (output) file.

For each product, create a templated section

Ideally we'd end up with Something like:

* product
** direct inputs
- input and amount
- input and amount
** raw ingredients
+ raw ingredient and amount
+ raw ingredient and amount

Parse the csv file:

For each line:

1. First column becomes the top header
2. Insert second header
3. Insert each ingredient and its amount as a separate item

Parse the output.org file, filling it out recursively

Note: copied over to another section, this stays here for later.

1. Open .org (output file)

2. Loop over output.org:

   1. Find product section.
   2. Find (next) ingredient lines in this product section.
   3. Pass the product and each direct input item and its number to raw-ingredients. (We can distinguish direct inputs from raw ingredients easily because org-mode supports multiple characters for defining lists. So we can just look for lines beginning with - and not really think about anything else.) This would look something like:

(raw-ingredients iron-gear-wheel iron-plate 2)
(raw-ingredients yellow-belt iron-plate 1)

1. (raw-ingredients (product item number)):

   1. Store in variables for clarity:

      • product
      • item
      • number

   2. Read .csv file:

      1. If a recipe for this item exists in the csv file:

         1. Go to the line with the recipe (first column.
         2. For each (item number) pair, call (raw-ingredients (product new-item (* new-number old-number))).

   3. If a recipe does not exist:

      1. Find * product section in the .org file.

      2. If the ingredient item already exists:

         1. Add new number we just got to the existing number.
      3. Else: write new raw ingredient line and number in this section.

Approach 2: working directly in the csv files

Idea/ outline

This is not for analysis as much as helping Galdoc out with creating compat layers for Galdoc's Manufacturing.

Take above csv data dump from Factorio, then:

1. Figure out appropriate categories. This is the starting point:

   • Telescoping, (inserters, belts, things that reach)
   • Metalworking,
   • Plastic,
   • Wood,
   • Stone,
   • Glass,
   • Electronics.

   • And possibly in the future:

     • Motors,
     • Agriculture,
     • Chemicals,
     • Small Arms / Equipment.

This has to be done manually. The csv file already includes an empty first column, which lets us manually go over it and add the tags to each item.

1. Then pull the output template CSV file, which should have the following structure:

   item-category, item-name, amt1, catitem1, amt2, catitem2, amt3, catitem3, amt4, catitem4, ...

In this example, we create an arbitrary number of columns, based on the largest number of ingredients a recipe requires from each category.

For example, stack inserters would look like so:

item-category, item-name, amt1, telescoping1, amt2, metal2, amt3, electronics3, amt4, electronics4
telescoping, stack-inserter, 1, fast-inserter, 15, iron-gear-wheel, 15, electronic-circuit, 1, advanced-circuit

This structure allows for unambiguous selection of the relevant data from the csv file, for the following reasons:

1. Amounts and categories are paired by the matching number at the end of the column name.
2. These cannot be confused with the amounts themselves, because the column names include [a-zA-Z] characters.
3. The first column makes it easy to find whether an item belongs to a particular column or not.
4. Additionally, this way we can automate creating the output .csv template, since we can check what number of columns we need for each ingredient category.

Testing 1:

Get csv from Factorio

This needs to be run manually rn, will figure out an automatic way later? Maybe?

/c
local recipes = {}
for name, recipe in pairs(game.recipe_prototypes) do
    local products = {}
    local ingredients = {}
    for _, product in pairs(recipe.products) do
        local amount = product.amount or product.amount_min .. "-" .. product.amount_max
        products[#products+1] = product.name .. ">" .. amount
    end
    for _, ingredient in pairs(recipe.ingredients) do
        ingredients[#ingredients+1] = ingredient.name .. "<" .. ingredient.amount
    end
    recipes[#recipes+1] = "," .. table.concat(products, "+") .. "," .. name .. "," .. recipe.subgroup.name .. "," ..  table.concat(ingredients, "+")
end
game.write_file("products-all.csv", table.concat(recipes, "\n"), false)

This outputs the following format of csvfile:

prod1>amt1+prod2>amt2, recipe.name, recipe.subgroup, ingr1<amt1+ingr2<amt2...

Example outputs that we could use:

Sorted by recipes:

Galdoc's groups	recipe-name	subgroup	prod1	prod2	prod…	ingr1	ingr2	ingr…
metal	iron-plate		iron-plate:1			iron-ore:1
wood
plastic
glass
stone

Sorted by products:

• One line per product, each product getting its own copy of all the recipes that make it (gg slag).
• Each product gets assigned a clear category. No issues, no conflicts.
• Problem: differing amounts between the recipes, so amounts need to be stated in a sep. column.
• If we use an unusual separator symbol for fields with more than one value (e.g. long lists of ingredients), we can then parse them very easily and remove ambiguity with a simple [whitespace, comma or +] test. Then situations like 'tin' being found in 'tinned wire' won't be a problem.

Galdoc's groups	product	amount	subgroup	recipe	required building		ingr1	ingr2	ingr…
metal	iron-plate	1		iron-plate	stone-furnace+steel-furnace+electric-furnace		iron-ore:1
wood
plastic
glass
stone

Move csv to the right directory

Note: If you have this open in Emacs, move your cursor into the code block and hit C-c C-c to execute the code. It's very handy if you have commands that you use often.

mv ~/.factorio/script-output/products-all.csv ./data

Sort for easier human readability

Sort the csv file by recipe subgroups

This allows us to find groups of related processes easier.

cat ./data/recipes-all.csv | sort -k3 -t, | column --table -s, -o, > ./data/recipes-all-sorted-subgroups.csv

Sort the csv file by product name

This allows us to find similarly named items easier

cat ./data/products-all.csv | sort -k2 -t, | column --table -s, -o, > ./data/recipes-all-sorted-product.csv
cp ./data/recipes-all-sorted-product.csv ./data/intermediate.csv

intermediate.csv is the file we'll be using for further testing, because it groups up a lot of items by material, which is what we want.

From here on, all changes will take place on the basis of the intermediate.csv file, and outputs will be directed to output.csv. This will prevent time loss in case of a mistyped command.

Parse the intermetiate.csv file, filling it out recursively

open(my $in, "<", "./data/intermediate.csv") or die "Can't open intermediates.csv";
open(my $out, ">", "./data/raw.csv") or die "Can't open raw.csv";
local $| = 1;
raw_ingredients();

sub raw_ingredients () {
    $product = $ARGV[1];
        while (<$in>){
            @line = split(",", $_);
            if ($line[1] == $product) {
                print "ingredients $line[4]\n";
                @ingredients = split(/\+/, $line[4]);
                print "\t\@ingredients: @ingredients \n";
                foreach my $ingredient (@ingredients) {
                    print "\t\t\$ingredient: $ingredient\n";
                    raw_ingredients($ingredient);
                    print "\t\t\trecurring!  \$ingredient = $ingredient\n";
                }
            }
            else {
                continue;
            }
        }
}

Define raw ingredients by their category

We have the following categories we'll be assigning:

• Telescoping, (inserters, belts, things that reach)
• Metalworking,
• Plastic,
• Wood,
• Stone,
• Glass,
• Electronics,

At the same time, we have 2843 entries to deal with. That's a lot, so let's start with the raw ingredients, and assign those using the following rough rules: For this reason, it is important that we first cover the base materials.

1. If the item exists in vanilla as a raw ingredient, it should be treated as such in the compatibility layer as well. If it is made craftable by a mod, its ingredients must also be tagged as raw materials.
2. The ingredient in question may not have a recipe of its own. E.g. iron ore in vanilla Factorio, or stiratite in AngelBob. In this case, it is a raw ingredient.

Temp code:

our @patterns = (

    # 0 metal

 "m/[\-?,?s*]
 (bronze|iron|steel|copper|
 zinc|lithium|tungsten|titanium|tin|
 nickel|silver|platinum|manganese|
 lead|gold|aluminium|aluminum)
 [\-?,?\s*]/",

    # 1 wood

 "m/[\-?,?s*]
 (seedlings|wood|wooden)
 [\-?,?\s*]/",

    # 2 plastic

 "m/[\-?,?s*]
 (plastic)
 [\-?,?\s*]/",

    # 3 electronics

 "m/[\-?,?s*]
 (electronic)
 [\-?,?\s*]/",

    # 4 glass

 "m/[\-?,?s*]
 (glass)
 [\-?,?\s*]/",

    # 5 stone

"m/[\-?,?s*]
 (limestone|stone|slag|concrete|sand)
 [\-?,?\s*]/");

open(my $in, "<", "./data/intermediate.csv") or die "Can't open intermediates.csv";
open(my $out, ">", "./data/output.csv") or die "Can't open output.csv";


while (<$in>) {
    my @line = split(",", $_);

    if ($line[1] =~ "$patterns[0]") {
        print $out "metal @line";
        print "$patterns[0]";
        print "metal @line";
    }
    elsif ($line[1] =~ "$patterns[1]") {
        print $out "wood @line";
    }
    elsif ($line[1] =~ "$patterns[2]") {
        print $out "plastic @line";
    }
    elsif ($line[1] =~ "$patterns[3]") {
        print $out "electronics @line";
    }
    elsif ($line[1] =~ "$patterns[4]") {
        print $out "glass @line";
    }
    elsif ($line[1] =~ "$patterns[5]") {
        print $out "stone @line";
    }
    else {
        print $out "@line";
        print "@line";
    }
}
close $in or die "$in: $!";
close $out or die "$out: $!";

New Idea to be given thought

https://wiki.factorio.com/Console#Access_a_mod's_data

• It may be possible to access lua directly and pull data from there.

• Even if not, it is possible to run Factorio with the --data-dump argument, which makes it output all the data the game has into a json file.

  • This is actually really convenient, because doing calculations on json files is rather.
• It may be more effective, faster and accurate than pulling data from an exported CSV.
• Further investigation needed.
• The key problem: it would be very challenging for a human/ person to make modifications to this export as-is. It's then necessary to create another layer of processing to make the output human-readable.
• ON the other hand, we don't yet have a system for importing recipes, items, and tech levels from a csv into Factorio mods…

• That may be something to explore to better define the constraints on this project.

  E.g. If it were possible to build the Lua files programmatically from user-created CSVs… Factorio modding would become significantly more accessible and easy for a ton of people. Of course, this would not be as powerful as diving into the code, however… I do see potential here with regards to large overhaul mods.

  Imagine:

  1. Export factorio base to csv.
  2. Change csv however you like, adding items, changing values, adding recipes, technologies, etc.
  3. Run a script which compiles your csv into a mod
  4. Have your mod running.

  There are several issues which bear addressing:

  1. Icons - in the case of a lack of icons, the script should default to something, so the GUI isn't broken.
  2. The script would have to dynamically generate a lot of lua code, or at least wrap the contents of the CSV in lua code. This may create licensing issues? Maybe?
  3. The item descriptions could be generated (partially) automatically, but they would still require someone to write large parts of them.
  4. Localization; it may be possible to also do with the script (eg hooking into google translate, deepl or sth), at least for the item names.

Elaboration:

The way I see this working is:

1. Parse mod lua files externally, independent from Factorio.
2. This creates a csv file including everything in a format that is both human-readable, and reversible back into Lua code.

Approach 3: pulling from Factorio data dump json

Get files into working area

• Get data dump from Factorio
• Copy data dump over to our working directory.

factorio --data-dump
cp ~/.factorio/script-output/data-raw-dump.json ./data/

Testing: Perl script parsing JSON into

Init:

use utf8;
use strict;
use warnings;

use JSON;
use Data::Dumper;

my $in;
{
    open(my $file, "<", "./data/data-raw-dump.json") or die "Can't open data-raw-dump.json";
    local $/;
    $in = <$file>;
    close $file;
}

my $data = decode_json($in);

# my $data = @($data_raw->(data)(children));

print Dumper($data);

my @list = ($data);

print "$data";

# foreach my $values (@list){
#     foreach my $value (@$values) {
#         print "Value =\t\tValue:\t" "$value->{?}->{'subgroup'}" "\n";
#     }
#             }

# open(my $out, ">", "./data/out-json-test.json") or die "Can't open out-json-test.json";
# close $out or die "$out: $!";

Run the script:

perl fra-json-test.pl

#+RESULTS: