Using syn to add functionality to progenitor-generated code.

1. The wages of syn is ...
by @jayvdb
John Mark Vandenberg

2. Chapter 10
https://www.google.com.au/books/edition/The_Communion_of_Saincts_A_Treatise_of_t/fR9lAAAAcAAJ

3. Romans 6:23
https://biblehub.com/interlinear/romans/6-23.htm

4. Romans 6:23
●
Syn:

5. Romans 6:23
●
Syn:

6. Romans 6:23
●
ſyn?

7. Romans 6:23
●
ſyn?

8. Romans 6:23
●
ſyn?

9. Romans 6:23
●
ſyn?

10. Romans 6:23 Rust crates.io

11. Romans 6:23 Rust crates.io

12. Java, with mustache templates,
with JavaScript CLI wrapper, and
does not support adding custom
“derive” to the structs, or easily
using custom templates
openapi-generator
Rust, but only supports OpenAPI
v2. Downgrading v3 to v2 is lossy,
but acceptable for models-only.
A small python script used to
downgrade.
No templating for custom derive.
paperclip
Rust, OpenAPI v3, YAML, Jinja
templates.
Builds, works well, but the
templates are not as good as
progenitor.
schema-tools
Rust, OpenAPI v3, no YAML.
Two pulls requests and we’re
100% Rust, OpenAPI v3, YAML.
Custom derive require hacks.
progenitor
01
01
02
02
03
03
04
04

13. Java, with mustache templates,
with JavaScript CLI wrapper, and
does not support adding custom
“derive” to the structs, or easily
using custom templates
openapi-generator
Rust, but only supports OpenAPI
v2. Downgrading v3 to v2 is lossy,
but acceptable for models-only.
A small python script used to
downgrade.
No templating for custom derive.
paperclip
Rust, OpenAPI v3, YAML, Jinja
templates.
Builds, works well, but refuses to
install.
schema-tools
Rust, OpenAPI v3, no YAML.
Two pulls requests and we’re
100% Rust, OpenAPI v3, YAML.
Custom derive require hacks.
progenitor
01
01
02
02
03
03
04
04

14. What is Progenitor?
Progenitor is a Rust crate for generating opinionated
clients from API descriptions in the OpenAPI 3.0.x
specification. It makes use of Rust futures for async API
calls and Streams for paginated interfaces.
It generates a type called Client with methods that
correspond to the operations specified in the OpenAPI
document.

15. Why use Progenitor?
OpenAPI between services:
* Standards based
* Keeps use honest
* Free code

16. Progenitor before syn – forking hell

17. Progenitor before syn – forking hell

18. Progenitor before syn – forking hell

19. Progenitor before syn – forking hell

20. Progenitor before syn – forking hell

21. Progenitor before syn – forking hell

22. Progenitor before syn – forking hell

23. fn main() {
let mut project_root = lets_find_up::find_up("Makefile.toml").unwrap().unwrap();
project_root.pop(); // Drop filename from path
let input_yaml = format!("{}/openapi.yaml", project_root.display());
let spec = load_spec(input_yaml);
// The use of `with_patch` here is to provide similar behaviour to
// unmerged https://github.com/oxidecomputer/typify/pull/184
let mut schema_names: ;
for schema_name in schema_names.iter().cloned() {
if !is_pascal_case(&schema_name) {
panic!("{schema_name} is not pascal case which will cause bugs.");
}
let mut patch = progenitor_impl::TypePatch::default();
// The enums have a Default impl added by typify
if !schema_name.ends_with("Enum")
// MySpecialEnum has a custom Default impl added in elsewhere.
&& schema_name != "MySpecialEnum"
{
patch.with_derive("Default");
}
if !TABLE_SCHEMAS.contains(&schema_name.as_str()) {
patch.with_derive("FieldType");
}
settings.with_patch(schema_name, &patch);
}
let input = generate_rust(&spec, &settings);
Progenitor after syn – lib-erty
...

24. fn main() {
let input = generate_rust(&spec, &settings);
let modified = transform_rust(input);
let out_filename = format!("{}/rust/models/src/lib.rs", project_root.display());
write_rust(out_filename, &modified);
}
Progenitor after syn – lib-erty
...
fn generate_rust(
spec: &OpenAPI,
settings: &progenitor_impl::GenerationSettings
) -> syn::File {
let mut generator = progenitor_impl::Generator::new(settings);
let original_tokens = generator.generate_tokens(spec).unwrap();
let ast: syn::File = syn::parse2(original_tokens).unwrap();
ast
}

25. Progenitor after syn – lib-erty
fn transform_rust(input: syn::File) -> syn::File {
let extra_butane_use: syn::Stmt = parse_quote! {
use butane::{
butane_type, model, ButaneJson, DataObject, FieldType, ForeignKey,
FromSql, Many, SqlType, SqlVal, SqlValRef, ToSql,
};
};
let extra_fake_use: syn::Stmt = parse_quote! {
use fake::{Dummy, Fake};
};
let extra_rand_use: syn::Stmt = parse_quote! {
use rand;
};
let use_butane_item: &syn::Item = cast!(&extra_butane_use, syn::Stmt::Item);
let use_fake_item: &syn::Item = cast!(&extra_fake_use, syn::Stmt::Item);
let use_rand_item: &syn::Item = cast!(&extra_rand_use, syn::Stmt::Item);
// Only the 'pub mod types { .. }' item from the input is wanted.
// i.e. remove the reqwest client
const MOD_TYPES_INDEX: usize = 3;
let mut mod_types: syn::ItemMod = cast!(&input.items[MOD_TYPES_INDEX], syn::Item::Mod).clone();
let (brace, mod_types_items) = mod_types.content.clone().unwrap();
let mut new_mod_types_items: Vec<syn::Item> = vec![];
new_mod_types_items.push(use_butane_item.clone());
new_mod_types_items.push(use_fake_item.clone());
new_mod_types_items.push(use_rand_item.clone());
for item in mod_types_items {
match item.clone() {
syn::Item::Struct(s) => {
let ident_name = s.ident.to_string();
if TABLE_SCHEMAS.contains(&ident_name.as_str()) {
// This branch is for the tables
let new_struct = transform_table(ident_name, s);
new_mod_types_items.push(syn::Item::Struct(new_struct));
} else {
// Serialise non-table schemas as JSON
let attribute: syn::Attribute = parse_quote! {
#[butane_type(Json)]
};
let mut new_item_body = s.clone();
new_item_body.attrs.insert(0, attribute);
new_mod_types_items.push(syn::Item::Struct(new_item_body));
}
}
syn::Item::Enum(e) => {
let ident_name = e.ident.to_string();
let mut new_item_body = e.clone();
// Serialise enums as JSON
let attribute: syn::Attribute = parse_quote! {
#[butane_type(Json)]
};
new_item_body.attrs.insert(0, attribute);
new_mod_types_items.push(syn::Item::Enum(new_item_body));
}
_ => new_mod_types_items.push(item),
}
}
mod_types.content = Some((brace, new_mod_types_items));
let pub_mod_migrations: syn::Stmt = parse_quote! {
pub mod butane_migrations;
};
let pub_mod_migrations: syn::Item = cast!(&pub_mod_migrations, syn::Stmt::Item).to_owned();
let new_item = syn::Item::Mod(mod_types);
let mut modified = input;
modified.items = vec![pub_mod_migrations, new_item];
modified
}

26. Progenitor after syn – lib-erty
fn transform_table(struct_name: String, input: syn::ItemStruct) -> syn::ItemStruct {
let mut new_struct = input;
for field in new_struct.fields.iter_mut() {
let name = field.ident.as_ref().unwrap().to_string();
if name.replace('_', "") == format!("{}id", struct_name.to_lowercase()) {
// This branch is for the primary key field
let attribute: syn::Attribute = parse_quote! {
#[pk]
};
field.attrs.insert(0, attribute);
} else {
// Any other field
match field.ty.clone() {
syn::Type::Path(p) => {
let type_name = p.path.segments[0].ident.to_string();
if type_name == "Vec" {
// If the Vec is of another table, replace Vec with Many.
// Currently this does not check if the target is a table.
// Also remove the `skip_serializing_if`.
// See https://github.com/Electron100/butane/issues/31
let mut new_field_type = p.clone();
let attribute: syn::Attribute = parse_quote! {
#[serde(default)]
};
field.attrs = vec![attribute];
new_field_type.path.segments[0].ident =
syn::Ident::new("Many", Span::call_site());
field.ty = syn::Type::Path(new_field_type);
} else if type_name == "Option" {
// Replace Option<T> with <Option<ForeignKey<T>>
// if T is another table.
let target: syn::AngleBracketedGenericArguments = cast!(
p.path.segments[0].arguments.clone(),
syn::PathArguments::AngleBracketed
);
let target: syn::Type =
cast!(target.args[0].clone(), syn::GenericArgument::Type);
let target: syn::TypePath = cast!(target, syn::Type::Path);
let target_type_name = target.path.segments[0].ident.to_string();
if TABLE_SCHEMAS.contains(&target_type_name.as_str()) {
// found reference to another table
let new_field_type: syn::TypePath = parse_quote! {
Option<ForeignKey<#target>>
};
field.ty = syn::Type::Path(new_field_type);
}
}
}
_ => {
panic!("cargo:warning=non path type {:?} {:?}nn", name, field.ty);
}
}
}
}

27. Progenitor after syn – lib-erty
fn transform_table(struct_name: String, input: syn::ItemStruct) ->
syn::ItemStruct {
let mut new_struct = input;
for field in new_struct.fields.iter_mut() {
let name = field.ident.as_ref().unwrap().to_string();
if name.replace('_', "") == format!("{}id", struct_name.to_lowercase())
{
// This branch is for the primary key field
let attribute: syn::Attribute = parse_quote! {
#[pk]
};
field.attrs.insert(0, attribute);
} else {
}
}
let attribute: syn::Attribute = parse_quote! {
#[model]
};
new_struct.attrs.insert(0, attribute);
new_struct
}
...

28. Progenitor after syn – sample code
#[model]
#[derive(Clone, Debug, Default, Deserialize, Dummy, PartialEq,
Serialize, actix_default_responder::JsonResponder,
utoipa::ToSchema)]
pub struct Patient {
#[serde(default, skip_serializing_if = "Option::is_none")]
pub age: Option<i64>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub dob: Option<chrono::DateTime<chrono::offset::Utc>>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub name: Option<String>,
#[pk]
pub patient_id: uuid::Uuid,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub sex: Option<PatientSexEnum>,
}

29. The wages of syn is ...

30. This work is licensed under
a Creative Commons Attribution-ShareAlike 3.0 Unported License.
It makes use of the works of
Kelly Loves Whales and Nick Merritt.