Florida International University, doctoral research
Inferential Discovery and Design Adaptation in Visual Generative Workflows
Research question
How does a designer’s relationship to their visual work in progress shift when inferential discovery can recur across a generative workflow?
In a generative workflow, model inference produces or transforms visual material, and that material can become grounds for the designer’s inference: the designer can take something visible in it to support a conclusion about the work in progress.
Where a workflow invokes model inference recurrently rather than episodically, the work can change through model inference at more points in the process, and the occasions on which the designer finds something consequential can recur. The possibility of recurrent inferential discovery makes this research an inquiry into deciding under an abundance of options.
The research examines how designers exercise agency in responding to inferential discovery across workflows that combine automated production with direct editing, compositing and physical making. It asks how the conditions of their work can enable or constrain what they can do.
Human inference is drawing a conclusion from available information and prior understanding. The word also names the conclusion drawn.
When information is limited, prior knowledge can shape which explanations a person considers and how plausible they find them (Tenenbaum and colleagues).
In his account of human reasoning with beliefs, Boghossian explains that a person draws a conclusion because they take other beliefs to support it. In design work, a designer can take something visible in the work as grounds for a conclusion about it.
In machine learning, inference applies a trained or fitted model to supplied input to produce a prediction or generated result.
In neural networks, training adjusts numerical weights and other parameters using data; inference processes the supplied input through the network using those learned values (Google machine learning glossary).
A text-to-image system, for example, uses a trained model to generate an image guided by a prompt (Rombach and colleagues). That image becomes visual material from which the designer can draw an inference.
In natural language processing, a trained model can perform a task called natural language inference by predicting whether a premise entails, contradicts, or is neutral toward a hypothesis (Bowman et al., 2015). Using inference for both people and models can encourage anthropomorphism when the shared term is taken to imply human reasoning or understanding. Following McDermott (1976) and Shanahan (2024), this account describes model inference through the mechanisms that produce its results.
In this research, an inference a designer finds becomes an inferential discovery as the designer determines what it means for adaptation and continuation of their work.
Design adaptation names the changes a discovery can prompt in the material being developed and the aims guiding it. It develops through situated judgments, and a judgment can shape the work the designer encounters in subsequent decisions, as when the designer prepares what the next generation receives (Suchman, 1987; Hutchins, 1995; Kirsh, 1995).
Here, abundance includes the information available in generated material, the possibility of further discovery, and the range of responses available across blended practice. A blended practice combines generative computation with other ways of making and editing, within one application or across several.
Simon (1971) observed that “a wealth of information creates a poverty of attention.” Abundance in this inquiry carries that cost: each generated result, each route through blended practice and each possible further discovery asks for judgment.
A designer can pursue something found in a generated result through another kind of operation. The response may develop the existing material, change the conditions of further production, or change the intention guiding the work. Further discovery can remain possible whichever response the designer chooses. That possibility can affect present decisions about continuation, as the designer considers what further work might reveal.
Further discovery need not depend on generating another output. Relationships can also become apparent through working with material already available. In his account of designing as reflective conversation with materials, Schön describes how designers’ moves can reveal unintended consequences and how their intentions can develop through the work. A discovery can change the designer’s understanding without being incorporated into a finished artifact.
More available material does not establish what matters to the designer. An undesirable result can occasion a consequential discovery.
Further discovery can occur after work has come to depend on an earlier decision. Reconsidering that decision can then affect the work that depends on it. What pursuing an interpretation requires depends partly on the possibilities for further work that remain after an operation. Those requirements can affect whether and how the designer continues.
Green and Blackwell call this resistance to change viscosity: changing one part of a working structure can require further changes to connected parts.
Continuation concerns what becomes of the work as the designer responds to it, including the possibility of deciding not to proceed. The work can be sufficient for its purpose while further discovery remains possible. A decision to stop can therefore rest on what the work accomplishes. Looking only at a finished output would leave much of the designer’s response and its consequences unexplained.
How the designer continues the work depends partly on their reasons and agency—the exercise of their capacity to act. The amount of automated production alone does not establish that capacity or explain how it is exercised.
The research has a technical component and a qualitative component: the Inference Resolution Map assesses software operations, and a visual workbook examines the designer’s response through evolving workflow examples.
4How does the map compare software operations?
The Inference Resolution Map assigns provisional X coordinates on a proposed scale of how trained or fitted prediction affects an operation’s returned result. Because what prediction determines depends on the task and its implementation, the map assesses named software tasks, provisionally defined here as operations. For each operation, the assessment identifies the software version, model where applicable, and settings relevant to its predictive task and computation.
For each operation, the assessment asks what the designer specifies, what specified computation determines, and what depends on prediction from a trained or fitted model in the returned result. The predictive task receives graded ratings describing what prediction determines in producing the returned result (scope), how much the supplied material and task leave the model to resolve (latitude), and which relationships it resolves together (coordination). X combines these ratings with implementation counts of the model’s parameters and computation. X is 1 when trained or fitted prediction makes no contribution to the returned result; where prediction contributes, X moves toward 0 as the ratings and transformed counts increase:
X = 1 − R[½(S + U + J3 × 5) + ½(p + c2)]
Ratings describe the task the prediction performs. Each is scored 0–5 against written anchors, and the support for a rating depends on the documentation, code, model files or papers available for the assessed implementation.
A workflow example connects the two components when it links the designer’s response and a working version to relevant operation assessments, including provisional X coordinates where supported. X summarizes the assessed operation’s predictive task and implementation; the accompanying account describes how the result is produced, as documented in the sources for that operation. The workbook describes what the designer finds significant and what responding requires, including what needs to be rebuilt when an earlier decision changes. For example, a generated clip may remain adjustable in timing and placement within an editable composition while its depicted movement stays fixed in the generated frames. If that composition is nested in another sequence, changing the clip’s timing changes when its movement occurs in that sequence. The designer may then find a different relationship between the movement and the next shot, prompting them to reconsider the earlier edit or what they want the sequence to achieve.
Shneiderman’s counterpoint also shapes the research’s direction. Greater automation need not be treated in advance as either a loss or a gain in the designer’s capacity to continue. A particular arrangement of model determination and retained controls may make one response possible while making another difficult. What that means is examined in relation to the work, the designer’s developing intentions and the response under consideration.
Three revision capabilities describe which responses an operation keeps possible. Shneiderman explicitly discusses interfaces that support revised intent and reversible actions. That supports a connection: these capabilities may describe part of the human control that Shneiderman keeps separate from automation. They make particular questions about further work available for examination:
Capability
What it helps investigate
Recovery and reuse
What earlier material remains accessible and usable when the designer considers another response.
Scope of available revision
Which aspects of the work the retained controls can change.
Preservation of dependent later work
What remains applied, and what needs to be rebuilt, when an earlier decision changes.
These descriptions can help explain control over revision across continuation without a second coordinate. They also give the research a particular concern: how earlier decisions and subsequent work remain connected when something is reconsidered. Considering these capabilities does not make preserving earlier decisions a goal in itself; their significance depends on what the designer is trying to pursue at that point.
R records whether a trained or fitted prediction affects the returned result. R is 1 when prediction from a trained or fitted model affects the returned result; a representation fitted to a photographed scene also counts as a fitted model. When implementation evidence establishes that the returned result is produced without trained or fitted prediction, R = 0 and X = 1. A programmed step that uses a prediction retains its contribution: a crop calculated from a predicted mask still depends on that prediction.
R · Prediction rules
A geometry or physical solver qualifies only if a trained or fitted prediction contributes. Unverified prediction dependence leaves R unresolved.
If a prediction is independently replaced before the result is returned, the replacement alone supplies that part of the result.
Foundry’s CopyCat documentation identifies the source image sequence and trained model file used to produce processed frames. Epic’s ML Deformer documentation identifies the supplied mesh, animation and target deformations used to train a model that approximates those deformations at runtime.
Autodesk’s AutoConstrain account identifies supplied sketch geometry and predicted constraints referring to that geometry. These training, prediction and workflow distinctions give R and the operation-level investigation a concrete basis: the assessment can identify what the model resolves.
Conventional software can still calculate selection membership, geometry or motion at this endpoint.
Which part of the result the prediction sets: settings, a selection, a component, a reconstruction of supplied material, new content within a restricted region, or content throughout.
S · rating scale
S concerns what is predicted and effectively restricted, rather than the percentage of pixels changed. A larger crop does not change a fixed component’s scope. Source protection requires evidence that processing preserves or reinstates the source information.
Rating
Definition
0
No in-scope learned determination remains in the returned result.
1
A learned category, scalar or bounded set of global settings; no dense field, structured relations or content construction is learned.
2
A learned selection, constraint/dimension set, coordinate or transformation field governing supplied content or geometry, without learned reconstruction of its appearance or new scene content.
3
A missing attribute channel or a fixed identified component/property is learned, while the surrounding observed instance defines the task: e.g. chroma, a face-recovery component or a driven mouth region. This functional restriction needs supporting evidence.
4
Reconstruct an observed instance or its fitted representation, or synthesize a completion under an effective restriction retaining source information outside the generative prediction. Reconstruction includes registered images and scene representations fitted to supplied observations. It does not assert exact pixel copying or unique recovery.
5
The operation can generate the entities/parts and arrangement throughout the output domain, with no documented protection/reinstatement restricting retained generative predictions to a component or region. A prompt or reference alone does not establish protection.
How much the supplied material and the task’s objective leave the model to resolve, from locating content already in the image, through faithful restoration, to constructing new content, including its arrangement, from a subject, style or action request or from surrounding context.
U · rating scale
U0 can still determine returned values: CopyCat can predict matte boundaries beyond the user-marked training examples. U follows the task definitions below; it does not count possible outputs or measure a probability distribution.
Rating
Definition
0
Recognize or locate observed content, or emulate a specified fixed transformation; no restoration, preference assignment or new-content task is included.
1
Retired from current coding. Fidelity-directed recovery is coded U2 regardless of sampling change; earlier U1 assessments are retained as history.
2
Recover an observed instance under a documented fidelity target, whether restoring corrupted samples on existing sampling support or estimating missing samples at increased sampling. A sampling change alone does not change U. Do not infer a fidelity target merely because training examples exist.
3
Infer an unobserved attribute, coordinate value, preference or design-intent relation among feasible choices not uniquely specified by the supplied instance and rules; no new semantic scene construction is the task.
Latitude 3 applies when the model chooses which of several observed candidates the result is about, with no location from the person and no class named by the operation; selection of a named class and selection the person marks stay at latitude 0.
4
Reconstruct plausible detail or appearance under a documented generative/perceptual/distributional role beyond recovery aimed at fidelity, or synthesize new content following an explicit supplied structural guide. Such a guide prescribes locations, shapes, poses, trajectories or event ordering for identifiable output parts, in words, coordinates or media; it is more than a high-level subject, style or action request. Evidence that the assessed mechanism uses it is needed. Guidance need not be hard protection. Predicting unobserved samples alone does not establish this anchor.
5
Construct a new content/configuration instance without an observed recovery target or an explicit supplied structural guide. Semantic instructions and surrounding context can constrain the task without specifying its internal arrangement.
ESRGAN §§3.1–3.3 supports distinguishing fidelity-oriented recovery from plausible-detail reconstruction. Latent diffusion §3.3 describes supplied conditioning of synthesis. These distinctions inform U’s task categories.
Which relationships the model resolves together: neighboring values, relationships across a supplied subject or scene, semantic and spatial relationships in synthesized content, or evolving relationships across video frames or camera views.
J · rating scale
Network connections show possible dependencies; subject or scene integration needs evidence that those dependencies operate across that subject or scene. A static fitted scene is rated by the relationships resolved in fitting it.
Rating
Definition
0
Separable learned responses; no learned coordination between locations, components or states is established as part of the task. Where the available evidence does not distinguish J0 from higher coordination levels, J remains unresolved.
1
Learned local neighboring values/features are resolved together.
2
Learned relationships across extended regions or scales of registered supplied content are resolved together.
Coordination 2 applies to fitted predictors whose features are the output's own spatial coordinates, linear fits included, when the fitting happens inside the operation whose returned result is assessed.
3
Integrate context across the supplied task instance or structure in learned predictions, including a face, a whole subject, a sketch or an observed scene. Establish this functional integration with product-linked task, mechanism or experimental evidence; an undisclosed architecture is not automatically zero. Record the reference domain.
4
The learned task jointly constructs semantic and spatial/structural relationships in synthesized content. Mere consistent appearance in a screenshot does not establish the mechanism.
5
A learned joint mechanism synthesizes evolving content relationships across time or views. Autoregressive internal tokens, conventional playback or repeated independent calls alone do not establish this.
Lumiere §3.1 describes a model that resolves relationships jointly across time, supporting the coordination distinction used by J.
P counts the distinct learned or fitted parameters in the model parts that affect the result.
P · Parameter counting rules
Parameters are numerical values learned during training or fitting, including weights and biases. Count contributing encoders, decoders and control networks; count shared parameters once, even when a component runs repeatedly. Exclude unused components and fixed nonlearned values.
Verify advertised model size against the parameter inventory; file size is not a parameter count. For a fitted scene, identify the scene and retained fitted state. P describes the implementation, not each parameter’s influence or the decisions delegated by the user.
The parameter reference is P₀ = 16,697,987 parameters, including biases, from the original ESRGAN generator, RRDBNet(3,3,64,23,gc=32). It has 351 convolution layers. The saved layer count records the inventory; the derivation script reproduces the count from the declared layer dimensions.
C counts the neural multiply–accumulates in matrix multiplications, convolutions and attention per returned RGB pixel, for a declared comparison workload and settings, including repeated sampling and guidance passes. A multiply–accumulate multiplies two values and adds the product to an accumulated total. Other output types, such as masks or 3D geometry, require a separately adopted computation basis.
C · Computation counting rules
Count contributing convolutions, matrix products and attention score/value products at their executed shapes and invocations, including repeated sampling and guidance branches. Divide by width × height for a still image, or width × height × returned frame count for video. One RGB pixel location counts once across its three channels.
Exclude training, conventional processing, bias additions, activations, normalization, softmax, interpolation and memory traffic. A contributing non-neural fitted mechanism can support R=1 while receiving C=0. Runtime, energy and GPU use require separate measures.
Record the comparison workload; prompt length has no separate weight. Without an adopted workload basis, the complete X remains unresolved. The RGB-pixel basis is not established for text, audio or geometric-model outputs.
For the original ESRGAN reference generator, enlarging a 512×512 RGB input to 2048×2048 takes 4,699,415,642,112 convolution MACs. Dividing by 2048² gives C₀ = 1,120,428 MACs per returned RGB pixel. The derivation script reproduces this arithmetic from the declared layer dimensions; the saved layer count records it.
When R = 1, the equation becomes:
X = 1 − S + U + J30task deduction − p + c4implementation deduction
The calculation starts at 1 and subtracts the two deductions. Larger deductions move the operation farther left on the map. Current coordinates use 0–1 units.
The first group divides the sum of three ratings, each with a maximum of 5, by 3 × 5. The second divides the sum of two transformed counts by 2. Multiplying each normalized group by ½ gives them equal coefficients. The task deduction can reach 0.5; the implementation deduction approaches 0.5 because p and c remain below 1 for finite counts. With R = 1 and the highest task ratings, X therefore approaches zero without reaching it.
With R = 1, each additional point in S, U or J subtracts 1/30 from X. Increasing S by one point and decreasing J by one point therefore leaves X unchanged. Addition lets a point of scope offset a point of coordination, so the same X can come from different ratings, and an operation’s ratings show which combination produced its X.
The equal weighting of the ratings and the counts is a development convention; empirical calibration would set the weights from data. Equal numerical steps between rating categories are also a convention.
The parameter and computation counts are converted into dimensionless, bounded values so they can be combined with the task ratings in X:
p = h(P/P0), c = h(C/C0)
h(z) = log2(1+z)1+log2(1+z)
A zero count becomes 0; a count equal to its reference becomes 0.5; larger counts produce values approaching 1. This transformation compresses large count differences. With R = 1, an increase of 0.1 in p or c subtracts 0.025 from X. Selecting ESRGAN as the reference and choosing this transformation are methodological conventions. An empirical midpoint of automation or a reference for another medium would need separate justification.
For the recorded GAN-trained ESRGAN reference in chaiNNer v0.25.1, the user supplies the image and selects the official RRDB_ESRGAN_x4.pth checkpoint. Reconstruction of the supplied image at native 4× enlargement supports S4. Its documented perceptual and adversarial objective supports U4. Convolutional reconstruction of neighboring image features supports the recorded J1. The prediction affects the returned image, so R = 1.
With RGB input without alpha, Custom Scale off and the whole image processed in one tile, P = P₀ and C = C₀, so p = c = 0.5:
X = 1 − 4 + 4 + 130 − 0.5 + 0.54
X = 1 − 0.30 − 0.25 = 0.45
Counts can also change while a network’s function stays equivalent, as when a network is enlarged or restructured (Chen, Goodfellow and Shlens, 2016; Ding et al., 2021), so whether parameter and computation counts should move X is under examination.
Category ordering and distances, substitution between criteria, weights and normalization remain under examination. A numerical difference in X does not by itself establish a difference in software determination.
X remains unresolved when a required value or an adopted computation basis is missing; an unknown value is not zero. X describes an operation’s processing; the percentage of human contribution, the designer’s agency and the probability of inferential discovery are separate questions. Following Shneiderman’s distinction, human control is taken up in the workbook rather than in X.
Supported processing findings and conditional calculations remain usable within their stated limits. A missing value prevents a functional judgment when it could change the applicable criterion.
Configuration rules
Model changes: reassess the operation when the selected model changes. Identical menus or approval buttons do not establish equivalent ratings.
Quality settings: establish whether the setting changes sampling, resolution, reasoning effort, models or other processing. Output quality is not itself a determination measure.
Auto settings: establish what the control selects for that operation. OpenRouter’s automatic model selection is one example. Automatic selection can inform D, the proposed account of procedural choices during use, which has no adopted weight.
Prompts: X is assessed for a stated operation and configuration, including the structural guidance its mechanism uses. A longer or shorter prompt to that operation and configuration keeps the same X.
The sources support developing X as a provisional comparative index. The numerical ratings remain methodological judgments.
Automation research has proposed scales like this before. Sheridan and Verplank set out ten levels from human control to computer automation, and Parasuraman, Sheridan and Wickens applied levels of automation to four stages of a task: acquiring information, analyzing it, choosing an action and carrying it out. They allow different automation levels for different functions within the same system, which helps establish why the particular operation and function being compared matter. These levels concern functions allocated to computers and their authority to suggest or execute actions, and they include rule-based automation. An operation is at X’s upper end when implementation evidence establishes that no trained or fitted prediction contributes to its returned result. Shneiderman objects that one line from human control to computer automation carries the message that “more automation meant less user control,” and separates the two into dimensions that can both be high.
Evidence
Relevance to the assessment
The measurement and composite-index literature
The OECD handbook recognizes both statistical and judgment-based choices in weighting and aggregation. It relates those choices to the intended comparison and examines their consequences through uncertainty and sensitivity analysis. Stevens distinguishes ordered categories from scales with equal intervals. Stevens, pp. 678–679, OECD handbook, pp. 31–35.
Parasuraman, Sheridan and Wickens; Shneiderman
These support examining distinct functions and levels within an operation, while keeping human control and agency separate from automation. They strengthen the rationale for retaining the operation’s profile alongside its coordinate. Parasuraman and colleagues, Shneiderman.
The contribution I am developing is a shared vocabulary for designers and a method for comparing operations, and for comparing designers’ responses to what those operations produce, so that claims about agency and automation can be examined against both technical evidence and documented visual practice.
The qualitative component is a visual workbook in Miro of evolving workflow examples. The examples connect supplied material and relevant settings with working versions and, where a discovery is documented, the designer’s account of it and what follows from the response. Through the examples, the research examines how the designer’s understanding, aims and judgments develop within the work, including what makes a result “coherent enough”: sufficient for its purpose, even when further discovery remains possible. Comparing the examples can test and refine the vocabulary for describing discoveries, what the available evidence supports about them, and the forms continuation takes. Where the records support a distinction the existing terms do not describe, the vocabulary is revised or extended.
The workbook accounts can record how intentions develop and state the criteria behind judgments that designers may otherwise make tacitly (Schön and Wiggins, 1992). The aim is for someone working on different material to take up, question and develop the distinctions and questions developed through the examples (Green and Blackwell, 1998; Gaver and Bowers, 2012; Yuan et al., 2026).
In practice, I have photographed my hands and edited reference images to convey relationships I had difficulty expressing through a text prompt alone. The workbook distinguishes what a prompt requests, what the operation’s mechanism supports, and what the designer then understands or values. That understanding can change what they prepare for the next operation.
Copper-figure example
The copper-figure example shows how the technical assessment could be connected to a designer’s response within a blended workflow, one that combines generative computation with other ways of making and editing. While developing the figure, I found that the head needed a bridge to the crown and considered a do-rag as a way to connect them. The process included generation, Photoshop Generative Fill, the Pen tool, layer masks and color grading. When generation failed to produce the required relationship, I constructed and composited the do-rag in Photoshop, then used the composite as the reference image for further generation. I accepted some loss of patina texture to retain the cultural fidelity I wanted in the do-rag.
References: the statue at night (left), the player in a black satin do-rag (center) and the championship trophy (right).
Working versions: braids over the statue’s carved hair (left) and cloth bands under the crown (center and right).
The figure with a black satin do-rag (left) and the do-rag in patina green (center). Right: a copper-toned night version with a cloth band under the crown, braids and the trophy.
Constructing the composite made the head–do-rag–crown relationship I had worked out part of the material supplied for further generation. This gives the technical assessment a specific question: how did the generative operation use that supplied relationship, and what did it determine in the returned material? Assessing that operation would require its software and model versions, relevant settings and technical evidence about how it uses reference material.
Test how independent assessors apply the written categories, using disagreements to refine the guidelines. Agreement would address consistency; category validity and X’s numerical calibration would require further evidence.
2024 — 2025
8Earlier research
Method history
The equation moved from aspect ratings to the combined method
August 2026 · Prototype catalogue of 417 rows, with coordinates proposed by research agents under broad numerical guidance.
20 September · Five-aspect specification score: X = ∑Si, with subject, composition, form, appearance and detail each scored 0–3 by how they are supplied.
21–27 September · Five opportunity-for-inference ratings: X = 15 − 3∑Oi.
29 September – 1 October · X/Y maps: X from the September assessments, with Y added for revision capacity.
Early October · Reference comparisons: X = XL + t(XH − XL), placing an operation between two reference operations.
4 October · Decision categories for 145 operations previously at X = 2: X = 2 + 3 × kept categories ÷ (applicable categories − 1).
14 October · S/U/J calibration proposal across 399 operations: X = 15 − 13(S + U + J)/15.
15 October · Combined method selected, adding model size and computation to the ratings: X = 15[1 − R((S + U + J)/30 + (p + c)/4)].
19 October · Output settings stop moving X; computation is counted at the implementation’s published default.
26 October · U1 retired; fidelity-directed recovery is coded U2 whatever the sampling change.
28 October – 3 November · The 15-point scale is dropped and X runs from 0 to 1: X = 1 − R[½(S + U + J)/(3 × 5) + ½(p + c)/2].
Earlier work examined how different input modalities supply partial spatial information, how generative systems resolve that information, and how people interpret the resulting structures.
While teaching architecture in Miami, I explored how people could learn to use tools through experiences tailored to them. My early research considered visualization and gamification as ways of making learning interactive, with climate education as an initial context. The emphasis subsequently moved toward personalized education and how learning could respond to individual needs.
As generative practice became a stronger focus, my attention shifted toward what designers encounter while working with these tools. The current dissertation focuses on the designer’s response to inferential discovery.
Research timeline
OCTOBER–NOVEMBER 2024
Visualization and interactive learning
Literature review and research-paper drafts examined 3D visualization, gamification, and AI-supported personalization in climate education.
DECEMBER 2024
Architectural education and AI
A Digital Futures paper considered prospective changes to architectural education, curricula, and relationships between educators and students.
MARCH 2025
Personalized learning
Personalized Learning in Higher Education: AI-Driven Analytics and Competency-Based Models as Catalysts for Reform placed adaptive learning and individual needs at the center of the inquiry.
9Personal reading & sources
Listening is an important part of how I learn and develop my work. My reading and listening span technology, creativity, perception, culture, and lived experience.
Linked titles and covers open their audiobook editions on Audible.
Ameisen, E., Lindsey, J., Pearce, A., Gurnee, W., Turner, N. L., Chen, B., Citro, C., Abrahams, D., Carter, S., Hosmer, B., Marcus, J., Sklar, M., Templeton, A., Bricken, T., McDougall, C., Cunningham, H., Henighan, T., Jermyn, A., Jones, A., … Batson, J. (2025). Circuit tracing: Revealing computational graphs in language models. Transformer Circuits Thread. https://transformer-circuits.pub/2025/attribution-graphs/methods.html
Anonymous authors. (n.d.). PSBench: Editing image via GUI agents in Photoshop [Manuscript submitted for publication; ICLR 2026 review version]. OpenReview. https://openreview.net/pdf?id=DWATchaS4U
Baltrušaitis, T., Ahuja, C., & Morency, L.-P. (2019). Multimodal machine learning: A survey and taxonomy. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(2), 423–443. https://doi.org/10.1109/TPAMI.2018.2798607
Bar-Tal, O., et al. (2024). Lumiere: A space-time diffusion model for video generation. ArXiv.org.https://arxiv.org/abs/2401.12945
Behrens, T. E. J., Muller, T. H., Whittington, J. C. R., Mark, S., Baram, A. B., Stachenfeld, K. L., & Kurth-Nelson, Z. (2018). What is a cognitive map? Organizing knowledge for flexible behavior. Neuron, 100(2), 490–509. https://doi.org/10.1016/j.neuron.2018.10.002
Benharrak, K., & Pavel, A. (2025). HistoryPalette: Supporting Exploration and Reuse of Past Alternatives in Image Generation and Editing [Version 1; Preprint]. arXiv. https://arxiv.org/abs/2501.04163v1
Berni, A., Borgianni, Y., Rotini, F., Gonçalves, M., & Thoring, K. (2024). Stimulating design ideation with artificial intelligence: Present and (short-term) future. Proceedings of the Design Society, 4, 1939–1948. https://doi.org/10.1017/pds.2024.196
Bowman, S. R., Angeli, G., Potts, C., & Manning, C. D. (2015). A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (pp. 632–642). Association for Computational Linguistics. https://doi.org/10.18653/v1/D15-1075
Bowker, G. C., & Star, S. L. (1999). Sorting things out: Classification and its consequences. MIT Press. MIT Press
Burke, A. (2025). Multimodal spatial intelligence [Unpublished doctoral project paper, ARC 7980]. Department of Architecture, Florida International University.
Caetano, I., Santos, L., & Leitão, A. (2020). Computational design in architecture: Defining parametric, generative, and algorithmic design. Frontiers of Architectural Research, 9(2), 287–300. https://doi.org/10.1016/j.foar.2019.12.008
Calefato, F., Castellano, G., & Rossano, V. (2019). RECODE: Revision control for digital images. Multimedia Tools and Applications, 78(23), 33169–33188. https://doi.org/10.1007/s11042-019-7735-9
Castro Pena, M. L., Carballal, A., Rodríguez-Fernández, N., Santos, I., & Romero, J. (2021). Artificial intelligence applied to conceptual design. A review of its use in architecture. Automation in Construction, 124, Article 103550. https://doi.org/10.1016/j.autcon.2021.103550
Chatzikyriakidis, S., & Lappin, S. (2026). Neuro-symbolic NLP: Taxonomy, assessment, and directions. Frontiers in Artificial Intelligence, 9, Article 1797587. https://doi.org/10.3389/frai.2026.1797587
Chen, T., Goodfellow, I., & Shlens, J. (2016). Net2Net: Accelerating learning via knowledge transfer. In Proceedings of the International Conference on Learning Representations. https://arxiv.org/abs/1511.05641
Chiriatti, M., Ganapini, M., Panai, E., Ubiali, M., & Riva, G. (2024). The case for human–AI interaction as system 0 thinking. Nature Human Behaviour, 8(10), 1829–1830. https://doi.org/10.1038/s41562-024-01995-5
Choi, J., Vasconcelos, H., Lee, H., Zou, C., Lee, T. Y., & Bernstein, M. (2026). FocusGen: Expanding visual design exploration with a simulated focus group of persona agents [Preprint]. arXiv. https://arxiv.org/abs/2608.28001
Christian, B., & Griffiths, T. (2016). Algorithms to live by: The computer science of human decisions. Henry Holt and Company. Macmillan / Henry Holt
Clark, A. (1997). Being there: Putting brain, body, and world together again. MIT Press. MIT Press
Conijn, R., Dux Speltz, E., van Zaanen, M., Van Waes, L., & Chukharev-Hudilainen, E. (2022). A product- and process-oriented tagset for revisions in writing. Written Communication, 39(1), 97–128. https://doi.org/10.1177/07410883211052104
Daston, L., & Galison, P. (2007). Objectivity. Zone Books. Zone Books
Deng, A., Cao, T., Chen, Z., & Hooi, B. (2025). Words or vision: Do vision-language models have blind faith in text? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. https://arxiv.org/abs/2503.02199
Doshi, A. R., & Hauser, O. P. (2024). Generative AI enhances individual creativity but reduces the collective diversity of novel content. Science Advances, 10(28), Article eadn5290. https://doi.org/10.1126/sciadv.adn5290
Doshi-Velez, F., & Kim, B. (2017). Towards a rigorous science of interpretable machine learning [Preprint]. arXiv. https://arxiv.org/abs/1702.08608
Dourish, P. (2001). Where the action is: The foundations of embodied interaction. MIT Press. MIT Press
Dow, S. P., Glassco, A., Kass, J., Schwarz, M., Schwartz, D. L., & Klemmer, S. R. (2010). Parallel prototyping leads to better design results, more divergence, and increased self-efficacy. ACM Transactions on Computer-Human Interaction, 17(4), Article 18. https://doi.org/10.1145/1879831.1879836
Driscoll, J. R., Sarnak, N., Sleator, D. D., & Tarjan, R. E. (1989). Making data structures persistent. Journal of Computer and System Sciences, 38(1), 86–124. https://doi.org/10.1016/0022-0000(89)90034-2
Fiebrink, R., Cook, P. R., & Trueman, D. (2011). Human model evaluation in interactive supervised learning. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 147–156). ACM. https://doi.org/10.1145/1978942.1978965
Ganz, F., & Müller, M. (2026). The weight is over: Interactive diffusion on consumer GPUs [Preprint]. arXiv. https://arxiv.org/abs/2609.21849
Garcin, S., Walker, T., McDonagh, S., Pearce, T., Bilen, H., He, T., Wang, K., & Bian, J. (2026). Beyond pixel histories: World models with persistent 3D state [Preprint]. arXiv. https://doi.org/10.48550/arXiv.2603.03482
Gharbi, M., Chen, J., Barron, J. T., Hasinoff, S. W., & Durand, F. (2017). Deep bilateral learning for real-time image enhancement. ACM Transactions on Graphics, 36(4), Article 118. https://doi.org/10.1145/3072959.3073592
Gilbert, D. T., & Ebert, J. E. J. (2002). Decisions and revisions: The affective forecasting of changeable outcomes. Journal of Personality and Social Psychology, 82(4), 503–514. https://doi.org/10.1037/0022-3514.82.4.503
Green, T. R. G., & Petre, M. (1996). Usability analysis of visual programming environments: A ‘cognitive dimensions’ framework. Journal of Visual Languages & Computing, 7(2), 131–174. https://doi.org/10.1006/jvlc.1996.0009
Hutchins, E. (1995). Cognition in the wild. MIT Press. MIT Press
Jaiswal, S., Prabhudesai, M., Bhardwaj, N., Qin, Z., Zadeh, A., Li, C., Fragkiadaki, K., & Pathak, D. (2026). Iterative refinement improves compositional image generation [Preprint]. arXiv. https://arxiv.org/abs/2601.15286
Jewitt, C. (Ed.). (2009). The Routledge handbook of multimodal analysis. Routledge. Routledge, first edition
Kautz, H. (2022). The third AI summer: AAAI Robert S. Engelmore Memorial Lecture. AI Magazine, 43(1), 105–125. https://doi.org/10.1002/aaai.12036
Kim, J., Pratt, M. J., Iyer, R. G., & Sriram, R. D. (2008). Standardized data exchange of CAD models with design intent. Computer-Aided Design, 40(7), 760–777. https://doi.org/10.1016/j.cad.2007.06.014
Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A. C., Lo, W.-Y., Dollár, P., & Girshick, R. (2023). Segment anything [Preprint]. arXiv. https://arxiv.org/abs/2304.02643
Kirsh, D. (1995). The intelligent use of space. Artificial Intelligence, 73(1–2), 31–68. DOI / Artificial Intelligence
Kress, G., & van Leeuwen, T. (2001). Multimodal discourse: The modes and media of contemporary communication. Arnold. Bloomsbury, original 2001 work
Lee, H.-P., Sarkar, A., Tankelevitch, L., Drosos, I., Rintel, S., Banks, R., & Wilson, N. (2025). The impact of generative AI on critical thinking: Self-reported reductions in cognitive effort and confidence effects from a survey of knowledge workers. In Proceedings of the CHI Conference on Human Factors in Computing Systems. ACM. https://doi.org/10.1145/3706598.3713778
Li, B., Lin, Z., Shechtman, E., Li, D., Fu, H., & Zhou, Y. (2026). StreamStudio: Exploring video authoring with streaming autoregressive generation [Preprint]. arXiv. https://arxiv.org/abs/2610.38467
Liapis, A., Yannakakis, G. N., Alexopoulos, C., & Lopes, P. (2016). Can computers foster human users' creativity? Theory and praxis of mixed-initiative co-creativity. Digital Culture & Education, 8(2), 136–153. Digital Culture & Education
Lin, Z., Ehsan, U., Agarwal, R., Dani, S., Vashishth, V., & Riedl, M. (2023). Beyond prompts: Exploring the design space of mixed-initiative co-creativity systems [Preprint]. arXiv. https://arxiv.org/abs/2305.07465
Ma, N., Tong, S., Jia, H., Hu, H., Su, Y.-C., Zhang, M., Yang, X., Li, Y., Jaakkola, T., Jia, X., & Xie, S. (2025). Inference-time scaling for diffusion models beyond scaling denoising steps [Preprint]. arXiv. https://arxiv.org/abs/2501.09732
Maher, M. L., & Poon, J. (1996). Modeling design exploration as co-evolution. Microcomputers in Civil Engineering, 11(3), 195–209. DOI / published 1996 article
Moors, A., & De Houwer, J. (2006). Automaticity: A theoretical and conceptual analysis. Psychological Bulletin, 132(2), 297–326. https://doi.org/10.1037/0033-2909.132.2.297
Myers, B. A., Lai, A., Le, T. M., Yoon, Y., Faulring, A., & Brandt, J. (2015). Selective undo support for painting applications. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. ACM. https://doi.org/10.1145/2702123.2702543
Naik, S., Shukla, P., Obi, I., Backus, J., Rasche, N., & Parsons, P. (2025). Tracing the invisible: Understanding students' judgment in AI-supported design work. In Creativity and Cognition (C&C '25). ACM. https://doi.org/10.1145/3698061.3734399
Nguyen, T., Michaels, J., Fiterau, M., & Jensen, D. (2025). Challenges in understanding modality conflict in vision-language models [Preprint]. arXiv. https://doi.org/10.48550/arXiv.2509.02805
Nooralahzadeh, F., Rohanian, O., Zhang, Y., Fürst, J., & Stockinger, K. (2026). Arbitration failure, not perceptual blindness: How vision-language models resolve visual-linguistic conflicts [Preprint]. arXiv. https://doi.org/10.48550/arXiv.2604.09364
OECD, European Union, & European Commission, Joint Research Centre. (2008). Handbook on constructing composite indicators: Methodology and user guide. OECD Publishing. https://doi.org/10.1787/9789264043466-en
Parasuraman, R., Sheridan, T. B., & Wickens, C. D. (2000). A model for types and levels of human interaction with automation. IEEE Transactions on Systems, Man, and Cybernetics—Part A: Systems and Humans, 30(3), 286–297. https://doi.org/10.1109/3468.844354
Park, J., Jang, K. J., Alasaly, B., Mopidevi, S., Zolensky, A., Eaton, E., Lee, I., & Johnson, K. (2025). Assessing modality bias in video question answering benchmarks with multimodal large language models. Proceedings of the AAAI Conference on Artificial Intelligence, 39(19), 19821–19829. https://doi.org/10.1609/aaai.v39i19.34183
Peirce, C. S. (1934). Collected papers of Charles Sanders Peirce: Vol. 5. Pragmatism and pragmaticism (C. Hartshorne & P. Weiss, Eds.). Harvard University Press.
Peng, X., Haldar, D., Mackay, W. E., & Koch, J. (2026). DesignTrace: Exploring, iterating and tracking design alternatives with GenAI. In Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems (pp. 1–22). ACM. https://doi.org/10.1145/3772318.3791036
Poole, D. L., & Mackworth, A. K. (2023). Artificial intelligence: Foundations of computational agents (3rd ed.). Cambridge University Press. https://artint.info/3e/html/ArtInt3e.html
Rezwana, J., & Maher, M. L. (2023). Designing creative AI partners with COFI: A framework for modeling interaction in human-AI co-creative systems. ACM Transactions on Computer-Human Interaction, 30(5), 1–28. https://doi.org/10.1145/3519026
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022). High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 10684–10695). https://doi.org/10.1109/CVPR52688.2022.01042
Roozenburg, N. F. M. (1993). On the pattern of reasoning in innovative design. Design Studies, 14(1), 4–18. DOI / Design Studies
Sajjadi, M. S. M., et al. (2022). Scene representation transformer: Geometry-free novel view synthesis through set-latent scene representations. CVPR. https://srt-paper.github.io
Schön, D. A. (1992). Designing as reflective conversation with the materials of a design situation. Research in Engineering Design, 3, 131–147. https://doi.org/10.1007/BF01580516
Shanahan, M. (2024). Still “Talking about large language models”: Some clarifications [Preprint]. arXiv. https://arxiv.org/abs/2412.10291
Shanahan, M. (2024). Talking about large language models. Communications of the ACM, 67(2), 68–79. https://doi.org/10.1145/3624724
Shen, H., Shen, L., Wu, W., & Zhang, K. (2025). IdeationWeb: Tracking the evolution of design ideas in human-AI co-creation. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems (pp. 1–19). ACM. https://doi.org/10.1145/3706598.3713375
Sheridan, T. B., & Verplank, W. L. (1978). Human and computer control of undersea teleoperators. Man-Machine Systems Laboratory, Massachusetts Institute of Technology. NTIS government report record
Shin, J., Li, Z., Zhang, R., Zhu, J.-Y., Park, J., Shechtman, E., & Huang, X. (2026). MotionStream: Real-time video generation with interactive motion controls. In International Conference on Learning Representations. https://arxiv.org/abs/2511.01266
Sim, M. Y., Zhang, W. E., Dai, X., & Fang, B. (2025). Can VLMs actually see and read? A survey on modality collapse in vision-language models. In Findings of the Association for Computational Linguistics: ACL 2025 (pp. 24452–24470). Association for Computational Linguistics. https://aclanthology.org/2025.findings-acl.1256/
Simon, H. A. (1955). A behavioral model of rational choice. The Quarterly Journal of Economics, 69(1), 99–118. https://doi.org/10.2307/1884852
Simon, H. A. (1971). Designing organizations for an information-rich world. In M. Greenberger (Ed.), Computers, communications, and the public interest (pp. 37–72). Johns Hopkins Press. Published chapter, Purdue-hosted scan
Sommers, N. (1980). Revision strategies of student writers and experienced adult writers. College Composition and Communication, 31(4), 378–388. https://www.jstor.org/stable/356588
Suchman, L. A. (1987). Plans and situated actions: The problem of human-machine communication. Cambridge University Press. Google Books, original 1987 work
Sun, Y., Jang, E., Ma, F., & Wang, T. (2024). Generative AI in the wild: Prospects, challenges, and strategies. In Proceedings of the CHI Conference on Human Factors in Computing Systems (pp. 1–16). ACM. https://arxiv.org/abs/2404.04101
Suwa, M., Gero, J. S., & Purcell, T. (2000). Unexpected discoveries and S-invention of design requirements: Important vehicles for a design process. Design Studies, 21(6), 539–567. https://doi.org/10.1016/S0142-694X(99)00034-4
Suwa, M., Gero, J. S., & Purcell, T. A. (1998). The roles of sketches in early conceptual design processes. In Proceedings of the Twentieth Annual Conference of the Cognitive Science Society (pp. 1043–1048). Lawrence Erlbaum Associates. https://escholarship.org/uc/item/1g4564qk
Tanimoto, S. L. (2013). A perspective on the evolution of live programming. In 2013 1st International Workshop on Live Programming (LIVE) (pp. 31–34). IEEE. https://doi.org/10.1109/LIVE.2013.6617346
Tankelevitch, L., Glassman, E. L., He, J., Kittur, A., Lee, M., Palani, S., Sarkar, A., Ramos, G., Rogers, Y., & Subramonyam, H. (2025). Understanding, protecting, and augmenting human cognition with generative AI: A synthesis of the CHI 2025 Tools for Thought workshop [Preprint]. arXiv. https://arxiv.org/abs/2508.21036
Tankelevitch, L., Kewenig, V., Simkute, A., Scott, A. E., Sarkar, A., Sellen, A., & Rintel, S. (2024). The metacognitive demands and opportunities of generative AI. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems (Article 680, pp. 1–24). ACM. https://doi.org/10.1145/3613904.3642902
Tenenbaum, J. B., Kemp, C., Griffiths, T. L., & Goodman, N. D. (2011). How to grow a mind: Statistics, structure, and abstraction. Science, 331(6022), 1279–1285. https://doi.org/10.1126/science.1192788
Terry, M., & Mynatt, E. D. (2002). Recognizing creative needs in user interface design. In Proceedings of the 4th Conference on Creativity & Cognition (pp. 38–44). ACM. https://doi.org/10.1145/581710.581718
Terry, M., Mynatt, E. D., Nakakoji, K., & Yamamoto, Y. (2004). Variation in element and action: Supporting simultaneous development of alternative solutions. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 711–718). ACM. https://doi.org/10.1145/985692.985782
Tran Luciani, D., Löwgren, J., & Lundberg, J. (2020). Designing fine-grained interactions for automation in air traffic control. Cognition, Technology & Work, 22(4), 685–701. https://doi.org/10.1007/s10111-019-00598-9
Wadinambiarachchi, S., Kelly, R. M., Pareek, S., Zhou, Q., & Velloso, E. (2024). The effects of generative AI on design fixation and divergent thinking. In Proceedings of the CHI Conference on Human Factors in Computing Systems (pp. 1–18). ACM. https://doi.org/10.1145/3613904.3642919
Wadinambiarachchi, S., Waycott, J., Rogers, Y., & Wadley, G. (2025). Imagining design workflows in agentic AI futures [Preprint; 37th Australian Conference on Human-Computer Interaction]. arXiv. https://arxiv.org/abs/2509.20731
Wen, Y., Jain, N., Kirchenbauer, J., Goldblum, M., Geiping, J., & Goldstein, T. (2023). Hard prompts made easy: Gradient-based discrete optimization for prompt tuning and discovery. In Advances in Neural Information Processing Systems (Vol. 36). https://arxiv.org/abs/2302.03668
Yannakakis, G. N., Liapis, A., & Alexopoulos, C. (2014). Mixed-initiative co-creativity. In Proceedings of the 9th International Conference on the Foundations of Digital Games. University of Malta repository
Yin, S., Zhang, Z., Tang, Z., Gao, K., Xu, X., Yan, K., Li, J., Chen, Y., Chen, Y., Shum, H.-Y., Ni, L. M., Zhou, J., Lin, J., & Wu, C. (2025). Qwen-Image-Layered: Towards inherent editability via layer decomposition [Preprint]. arXiv. https://arxiv.org/abs/2512.15603
Yuan, D., Graham, C., Yakura, H., & Lim, A. (2026). Using the blackbox in embodied AI art practice: Uncertainty at the interface. In Designing Interactive Systems Conference (DIS ’26) (pp. 4181–4201). ACM. https://doi.org/10.1145/3800645.3813020
Yuille, A., & Kersten, D. (2006). Vision as Bayesian inference: Analysis by synthesis? Trends in Cognitive Sciences, 10(7), 301–308. https://doi.org/10.1016/j.tics.2006.05.002
Yun, J., Park, J., Lim, H., Hyung, J., Chung, H., & Choo, J. (2026). ReDesign: Recovering Editable Design Structures from Images via Agentic Decomposition [Preprint]. arXiv. https://arxiv.org/abs/2607.25565v1
Zhang, C., Ge, Q., & Han, T. (2026). Overwhelmed by AI? How task difficulty and creative potential moderate cognitive load in human-AI co-creation. In Extended Abstracts of the 2026 CHI Conference on Human Factors in Computing Systems (pp. 1–6). ACM. https://doi.org/10.1145/3772363.3798752
Zhang, K., Zuo, W., Chen, Y., Meng, D., & Zhang, L. (2016). Beyond a Gaussian denoiser: Residual learning of deep CNN for image denoising [Preprint]. arXiv. https://arxiv.org/abs/1608.03981
Zhang, Z., Zhou, W., Zhao, J., & Li, H. (2025). Robust multimodal large language models against modality conflict. In Proceedings of the 42nd International Conference on Machine Learning. https://arxiv.org/abs/2507.07151
Zhao, Z., Rees, C., Bailey, G., Jones, M., Robinson, S., & Pearson, J. (2026). Less redraw, more explore: Suggestion and completion for sketch-to-image. In Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems (pp. 1–11). ACM. https://doi.org/10.1145/3772318.3791026
Sources for map operations
Technical documentation and research supporting the operation records on this page.