Inhalt
Aktueller Ordner:
/📓 Jupyter Notebook - Kernel: Python (Pyodide)
📝 Markdown Zelle #1
Qualitative social research has overslept cognitivism.
So was missed, the reconstruction of latent structures of meaning
secured by the construction of generative rules in the sense of algorithms.
For valid category systems (cf. Mayring) can be algorithmic rules
specify a finite automaton
(vg. Buy, Paul.: ARS, Grammar-Induction, Parser, Grammar-Transduction).
Now, posthumanism, poststructuralism, and transhumanism are parasitizing Opaque AI.
And if they don't parasitize them, they are mutual symbionts.
Karl Popper is then replaced by Harry Potter and
qualitative social research and large language models become too little explanatory,
but impressive cargo-cult of a non-explanatory and everything
veiled postmodernism.
For the algorithmic recursive sequence analysis it was shown that
that for the record of a sequence of actions
at least one grammar can be specified
(Inductorin Scheme, Parser in Pascal, Transduktor in Lisp, see Koop, P.).
ARS is a qualitative procedure
that latent rules of recorded action sequences
can be refuted to reconstruct.
A large language model can be reprogrammed in such a way that it
determined categories of a qualitative content analysis (cf. Mayring)
can reconstruct.
However, the explanatory value of such a model is negligible,
because it just isn't explained.
To show this, the following
the post-programming of a large language model is described.
In [ ]:
From the corpus of encodings of a transcribed protocol, a deep language model can be used
a simulation of a sales talk can be run.
The algorithm of the deep language model then stands for the generative structure.
Provide good introductions:
Steinwender, J., Schwaiger, R.:
Neuronale Netze programmieren mit Python
2. Auflage 2020
ISBN 978-3-8362-7452-4
Trask, A. W.:
Neuronale Netze und Deep Learning kapieren
Der einfache Praxiseinstieg mit Beispielen in Python
1. Auflage 2020
ISBN 978-3-7475-0017-0
Hirschle, J.:
Deep Natural Language Processing
1. Auflage 2022
ISBN 978-3-446-47363-8
The data structures in this text are taken from the above title by A. IN. Trask reprogrammed.
The deep language model for sales talks is then derived from this.
Neural networks are multi-dimensional, mostly two-dimensional data fields of rational numbers.
A hidden layer of predictive weights weights the input layer data, propagates the results to the next layer, and so on,
until an open output layer then outputs them.
In the training phase, the weights are backpropagated, in the case of large language models with recurrent networks, with attention to the logged context.
The illustrative examples attempt to compare a team's scores by weighting the number of toes,
the games won so far and the number of fans to determine the future chances of winning.
📝 Markdown Zelle #2
Just an input date, here is the toe count:
🧩 Code Zelle #3
# The network
weight= 0.1
def neural_network (input, weight):
output= input* weight
return output
# Application of the network
number_of_toes= [8.5, 9.5, 10, 9]
input= number_of_toes[0]
output= neural_network (input, weight)
print(output)📝 Markdown Zelle #4
0.8500000000000001
📝 Markdown Zelle #5
Now with three input data (toe count, wins so far, number of fans):
🧩 Code Zelle #6
def propagation function(a,b):
assert(only== only (b))
output= 0
for i in range(len(s)):
output+= (a[i] * b[i])
return output
weight= [0.1, 0.2, 0]
def neural_network(input, weight):
output= propagationfunction(input,weight)
return output
toes= [8.5, 9.5, 9.9, 9.0]
winrate= [0.65, 0.8, 0.8, 0.9]
fans = [1.2, 1.3, 0.5, 1.0]
input= [toes[0],winrate[0],fans[0]]
output= neural_network(input,weight)
print(output)📝 Markdown Zelle #7
0.9800000000000001
📝 Markdown Zelle #8
Now with the library numy (arrays, vectors, matrices):
🧩 Code Zelle #9
import numpy as the
weight= the.array([0.1, 0.2, 0])
def neural_network(input, weight):
output= input.dots(weight)
return output
toes= the.array([8.5, 9.5, 9.9, 9.0])
winrate= the.array([0.65, 0.8, 0.8, 0.9])
fans = the.array([1.2, 1.3, 0.5, 1.0])
input= the.array([toes[0],winrate[0],fans[0]])
output= neural_network(input,weight)
print(output)📝 Markdown Zelle #10
0.9800000000000001
📝 Markdown Zelle #11
The weights can be adjusted until
until the error is minimized.
until the error is minimized.
🧩 Code Zelle #12
# Principle example
weight= 0.5
input= 0.5
desired_prediction= 0.8
increment= 0.001
for iteration in range(1101):
forecast= input* weight
mistake= (forecast- desired_prediction)** 2
print("Error:"+ str(error)+ "Prediction:"+ str(prediction))
higher_prediction= input* (weight+ increment)
deeper_error= (desired_prediction- higher_prediction)** 2
higher_prediction= input* (weight- increment)
deeper_bugs= (desired_prediction- deeper_prediction)** 2
if(deeper_error< higher_error):
weight= weight- increment
if(deeper_error> higher_error):
weight= weight+ increment🧩 Code Zelle #13
# Trask, A. W.:
# Understand neural networks and deep learning
# The simple introduction to practice with examples in Python
#1st Edition 2020
# ISBN 978-3-7475-0017-0
import numpy as e.g.
# Object class array
class Tensor (object):
def __init__(self,data,
autograd=False,
creators=None,
creation_op=None,
id=None):
self.data = e.g..array(data)
self.autograd = autograd
self.grad = None
if(id is None):
self.id = e.g..random.randint(0,1000000000)
else:
self.id = id
self.creators = creators
self.creation_op = creation_op
self.children = {}
if(creators is not None):
for c in creators:
if(self.id not in c.children):
c.children[self.id] = 1
else:
c.children[self.id] += 1
def all_children_grads_accounted_for(self):
for id,cnt in self.children.items():
if(cnt != 0):
return False
return True
def backward(self,grad=None, grad_origin=None):
if(self.autograd):
if(grad is None):
grad = Tensor(np.ones_like(self.data))
if(grad_origin is not None):
if(self.children[grad_origin.id] == 0):
return
print(self.id)
print(self.creation_op)
print(len(self.creators))
for c in self.creators:
print(c.creation_op)
raise Exception("cannot backprop more than once")
else:
self.children[grad_origin.id] -= 1
if(self.grad is None):
self.grad = grad
else:
self.grad += grad
assert grad.autograd == False
if(self.creators is not None and
(self.all_children_grads_accounted_for() or
grad_origin is None)):
if(self.creation_op == "add"):
self.creators[0].backward(self.grad, self)
self.creators[1].backward(self.grad, self)
if(self.creation_op == "sub"):
self.creators[0].backward(Tensor(self.grad.data), self)
self.creators[1].backward(Tensor(self.grad.__neg__().data), self)
if(self.creation_op == "I have"):
new = self.grad * self.creators[1]
self.creators[0].backward(new , self)
new = self.grad * self.creators[0]
self.creators[1].backward(new, self)
if(self.creation_op == "mm"):
c0= self.creators[0]
c1= self.creators[1]
new = self.grad.mm(c1.transpose())
c0.backward(new)
new = self.grad.transpose().mm(c0).transpose()
c1.backward(new)
if(self.creation_op == "transpose"):
self.creators[0].backward(self.grad.transpose())
if("sum" in self.creation_op):
dim = int(self.creation_op.split("_")[1])
self.creators[0].backward(self.grad.expand(dim,
self.creators[0].data.shape[dim]))
if("expand" in self.creation_op):
dim = int(self.creation_op.split("_")[1])
self.creators[0].backward(self.grad.am(dim))
if(self.creation_op == "neg"):
self.creators[0].backward(self.grad.__neg__())
if(self.creation_op == "sigmoid"):
ones = Tensor(np.ones_like(self.grad.data))
self.creators[0].backward(self.grad * (self * (ones - self)))
if(self.creation_op == "tanh"):
ones = Tensor(np.ones_like(self.grad.data))
self.creators[0].backward(self.grad * (ones - (self * self)))
if(self.creation_op == "index_select"):
new_grad = e.g..zeros_like(self.creators[0].data)
indices_ = self.index_select_indices.data.flatten()
grad_ = grad.data.reshape(len(indices_), -1)
for i in range(len(indices_)):
new_grad[indices_[i]] += grad_[i]
self.creators[0].backward(Tensor(new_grad))
if(self.creation_op == "cross_entropy"):
dx = self.softmax_output - self.target_dist
self.creators[0].backward(Tensor(dx))
def __add__(self, other):
if(self.autograd and other.autograd):
return Tensor(self.data + other.data,
autograd=True,
creators=[self,other],
creation_op="add")
return Tensor(self.data + other.data)
def __neg__(self):
if(self.autograd):
return Tensor(self.data * -1,
autograd=True,
creators=[self],
creation_op="neg")
return Tensor(self.data * -1)
def __sub__(self, other):
if(self.autograd and other.autograd):
return Tensor(self.data - other.data,
autograd=True,
creators=[self,other],
creation_op="sub")
return Tensor(self.data - other.data)
def __mul__(self, other):
if(self.autograd and other.autograd):
return Tensor(self.data * other.data,
autograd=True,
creators=[self,other],
creation_op="I have")
return Tensor(self.data * other.data)
def sum(self, dim):
if(self.autograd):
return Tensor(self.data.I am
autograd=True,
creators=[self],
creation_op="sum_"+str(dim))
return Tensor(self.data.am(dim))
def expand(self, dim,copies):
trans_cmd = list(range(0,len(self.data.shape)))
trans_cmd.insert(dim,len(self.data.shape))
new_data = self.data.repeat(copies).reshape(list(self.data.shape) + [copies]).transpose(trans_cmd)
if(self.autograd):
return Tensor(new_data,
autograd=True,
creators=[self],
creation_op="expand_"+str(dim))
return Tensor(new_data)
def transpose(self):
if(self.autograd):
return Tensor(self.data.transpose(),
autograd=True,
creators=[self],
creation_op="transpose")
return Tensor(self.data.transpose())
def mm(self, x):
if(self.autograd):
return Tensor(self.data.dot(x.data),
autograd=True,
creators=[self,x],
creation_op="mm")
return Tensor(self.data.dot(x.data))
def sigmoid(self):
if(self.autograd):
return Tensor(1 / (1 + e.g..exp(-self.data)),
autograd=True,
creators=[self],
creation_op="sigmoid")
return Tensor(1 / (1 + e.g..exp(-self.data)))
def tanh(self):
if(self.autograd):
return Tensor(np.fishy(self.data),
autograd=True,
creators=[self],
creation_op="tanh")
return Tensor(np.fishy(self.data))
def index_select(self, indices):
if(self.autograd):
new = Tensor(self.data[indices.data],
autograd=True,
creators=[self],
creation_op="index_select")
new.index_select_indices = indices
return new
return Tensor(self.data[indices.data])
def softmax(self):
temp = e.g..exp(self.data)
softmax_output = temp / e.g..sum(temp,
axis=len(self.data.shape)-1,
keepdims=True)
return softmax_output
def cross_entropy(self, target_indices):
temp = e.g..exp(self.data)
softmax_output = temp / e.g..sum(temp,
axis=len(self.data.shape)-1,
keepdims=True)
t = target_indices.data.flatten()
p = softmax_output.reshape(len(t),-1)
target_dist = e.g..eye(p.shape[1])[t]
loss = -(e.g.log(p) * (target_dist)).sum(1).mean()
if(self.autograd):
out = Tensor(loss,
autograd=True,
creators=[self],
creation_op="cross_entropy")
out.softmax_output = softmax_output
out.target_dist = target_dist
return out
return Tensor(loss)
def __repr__(self):
return str(self.data.__repr__())
def __str__(self):
return str(self.data.__str__())
class Layer(object):
def __init__(self):
self.parameters = list()
def get_parameters(self):
return self.parameters
class SGD(object):
def __init__(self, parameters, alpha=0.1):
self.parameters = parameters
self.alpha = alpha
def zero(self):
for p in self.parameters:
p.grad.data *= 0
def step(self, zero=True):
for p in self.parameters:
p.data -= p.grad.data * self.alpha
if(zero):
p.grad.data *= 0
class Linear(Layer):
def __init__(self, n_inputs, n_outputs, bias=True):
super().__heat__()
self.use_bias = bias
IN= e.g..random.randn(n_inputs, n_outputs) * e.g..sqrt(2.0/(n_inputs))
self.weight = Tensor(W, autograd=True)
if(self.use_bias):
self.bias = Tensor(np.zeros(n_outputs), autograd=True)
self.parameters.append(self.weight)
if(self.use_bias):
self.parameters.append(self.bias)
def forward(self, input):
if(self.use_bias):
return input.mm(self.weight)+self.bias.expand(0,len(input.data))
return input.mm(self.weight)
class Sequential(Layer):
def __init__(self, layers=list()):
super().__heat__()
self.layers = layers
def add(self, layer):
self.layers.append(layer)
def forward(self, input):
for layer in self.layers:
input = layer.forward(input)
return input
def get_parameters(self):
params = list()
for l in self.layers:
params += l.get_parameters()
return params
class Embedding(Layer):
def __init__(self, vocab_size, dim):
super().__heat__()
self.vocab_size = vocab_size
self.dim = dim
# this random initialiation style is just a convention from word2vec
self.weight = Tensor((np.random.rand(vocab_size, dim) - 0.5) / dim, autograd=True)
self.parameters.append(self.weight)
def forward(self, input):
return self.weight.index_select(input)
class Tanh(Layer):
def __init__(self):
super().__heat__()
def forward(self, input):
return input.tanh()
class Sigmoid(Layer):
def __init__(self):
super().__heat__()
def forward(self, input):
return input.sigmoid()
class CrossEntropyLoss(object):
def __init__(self):
super().__heat__()
def forward(self, input, target):
return input.cross_entropy(target)
# Sprachmodell Long Short Term Memory
class LSTMCell(Layer):
def __init__(self, n_inputs, n_hidden, n_output):
super().__heat__()
self.n_inputs = n_inputs
self.n_hidden = n_hidden
self.n_output = n_output
self.xf = Linear(n_inputs, n_hidden)
self.xi = Linear(n_inputs, n_hidden)
self.xo= Linear(n_inputs, n_hidden)
self.xc = Linear(n_inputs, n_hidden)
self.hf = Linear(n_hidden, n_hidden, bias=False)
self.hi = Linear(n_hidden, n_hidden, bias=False)
self.to= Linear(n_hidden, n_hidden, bias=False)
self.hc = Linear(n_hidden, n_hidden, bias=False)
self.w_ho= Linear(n_hidden, n_output, bias=False)
self.parameters += self.xf.get_parameters()
self.parameters += self.xi.get_parameters()
self.parameters += self.xo.get_parameters()
self.parameters += self.xc.get_parameters()
self.parameters += self.hf.get_parameters()
self.parameters += self.hi.get_parameters()
self.parameters += self.to.get_parameters()
self.parameters += self.hc.get_parameters()
self.parameters += self.w_ho.get_parameters()
def forward(self, input, hidden):
prev_hidden = hidden[0]
prev_cell = hidden[1]
f = (self.xf.forward(input) + self.hf.forward(prev_hidden)).sigmoid()
i = (self.xi.forward(input) + self.hi.forward(prev_hidden)).sigmoid()
O= (self.xo.forward(input) + self.to.forward(prev_hidden)).sigmoid()
g = (self.xc.forward(input) + self.hc.forward(prev_hidden)).tanh()
c = (f * prev_cell) + (i * g)
h = O* c.tanh()
output = self.w_ho.forward(h)
return output, (h, c)
def init_hidden(self, batch_size=1):
init_hidden = Tensor(np.zeros((batch_size,self.n_hidden)), autograd=True)
init_cell = Tensor(np.zeros((batch_size,self.n_hidden)), autograd=True)
init_hidden.data[:,0] += 1
init_cell.data[:,0] += 1
return (init_hidden, init_cell)
import sys,random,math
from collections import Counter
import numpy as e.g.
import sys
e.g..random.seed(0)
# Import the VKG BODY
f = open('VKGKORPUS.TXT','r')
raw = f.read()
f.close()
vocab = list(set(raw))
word2index = {}
for i,word in enumerate(vocab):
word2index[word]=i
indices = e.g..array(list(map(lambda x:word2index[x], raw)))
embed = Embedding(vocab_size=only (vocab), dim=512)
model = LSTMCell(n_inputs=512, n_hidden=512, n_output=just(vocab))
model.w_ho.weight.data *= 0
criterion = CrossEntropyLoss()
optimum= SGD(parameters=model.get_parameters() + embed.get_parameters(), alpha=0.05)
def generate_sample(n=30, init_char=' '):
s = ""
hidden = model.init_hidden(batch_size=1)
input = Tensor(np.array([word2index[init_char]]))
for i in range(n):
rnn_input = embed.forward(input)
output, hidden = model.forward(input=rnn_input, hidden=hidden)
# output.data *= 25
# temp_dist = output.softmax()
# temp_dist /= temp_dist.sum()
# m = (temp_dist > np.random.rand()).argmax()
m = output.data.argmax()
c = vocab[m]
input = Tensor(np.array([m]))
s += c
return s
batch_size = 16
bptt= 25
n_batches = int((index.shape[0] / (batch_size)))
trimmed_indices = indices[:n_batches*batch_size]
batched_indices = trimmed_indices.reshape(batch_size, n_batches).transpose()
input_batched_indices = batched_indices[0:-1]
target_batched_indices = batched_indices[1:]
n_bptt= int(((n_batches-1) / bptt))
input_batches = input_batched_indices[:n_bptt*bptt].reshape(n_bptt,bptt,batch_size)
target_batches = target_batched_indices[:n_bptt*bptt].reshape(n_bptt, bptt, batch_size)
min_loss = 1000
# Training of the neural network
def train(iterations=400):
for iter in range(iterations):
total_loss = 0
n_loss = 0
hidden = model.init_hidden(batch_size=batch_size)
batches_to_train = len(input_batches)
# batches_to_train = 32
for batch_i in range(batches_to_train):
hidden = (Tensor(hidden[0].data, autograd=True), Tensor(hidden[1].data, autograd=True))
losses = list()
for t in range(bptt):
input = Tensor(input_batches[batch_i][t], autograd=True)
rnn_input = embed.forward(input=input)
output, hidden = model.forward(input=rnn_input, hidden=hidden)
target = Tensor(target_batches[batch_i][t], autograd=True)
batch_loss = criterion.forward(output, target)
if(t == 0):
losses.append(batch_loss)
else:
losses.append(batch_loss + losses[-1])
loss = losses[-1]
loss.backward()
optimum.step()
total_loss += loss.data / bptt
epoch_loss = e.g..exp(total_loss / (batch_i+1))
min_loss =1000
if(epoch_loss < min_loss):
min_loss = epoch_loss
print()
log = "\r Iter:" + str(iter)
log += " - Alpha:" + str (opt.alpha)[0:5]
log += " - Batch "+str(batch_i+1)+"/"+str(len(input_batches))
log += " - Min Loss:" + str(min_loss)[0:5]
log += " - Loss:" + str(epoch_loss)
if(batch_i == 0):
log += " - " + generate_sample(n=70, init_char='T').replace("\n"," ")
if(batch_i % 1 == 0):
sys.stdout.write(log)
optimum.alpha *= 0.99
train(100)
def generate_sample(n=30, init_char=' '):
s = ""
hidden = model.init_hidden(batch_size=1)
input = Tensor(np.array([word2index[init_char]]))
for i in range(n):
rnn_input = embed.forward(input)
output, hidden = model.forward(input=rnn_input, hidden=hidden)
output.data *= 15
temp_dist = output.softmax()
temp_dist /= temp_dist.sum()
# m = (temp_dist > np.random.rand()).argmax() # sample from predictions
m = output.data.argmax() # take the max prediction
c = vocab[m]
input = Tensor(np.array([m]))
s += c
return s
print(generate_sample(n=500, init_char='\n'))🧩 Code Zelle #14
print(generate_sample(n=500, init_char='\n'))
📝 Markdown Zelle #15
Output of a generated example:
📝 Markdown Zelle #16
KBG VBG
KBBD VBBD KBA VBA KAE VAE KAA VAA
KBBD VBBD KBA VBA KBBD VBBD KBA VBA KBBD VBBD KBA VBA KAE VAE
KBG VBG
KBBD VBBD KBA VBA KAE VAE KAE VAE KAE VAE KAE VAE KAA VAA
KBBD VBBD KBA VBA KAE VAE KAE VAE KAA VAA
KBBD VBBD KBA VBA KAE VAE KAA VAA
KBBD VBBD KBA VBA KBBD VBBD KBA VBA KAE VAE KAA VAA
KBG VBG
KBBD KBA VBA KBBD VBBD KBA VBA KAE VAE KAE VAE KAA VAA
KBBD VBBD KBA VBA KBBD KBA VBA KBBD VBBD KBA VBA KBBD VBBD KBA
KAE VAE KAA VAA
KBBD VBBD KBA VBA KAE VAE KAE VAE VAE KAA VAA
KBBD VBBD KBA VBA KAE VAE KAA VAA
KBBD VBBD KBA VBA KBBD VBBD KBA VBA KBBD VBBD KBA VBA KAE VAE
KBG VBG
KBBD VBBD KBA VBA KAE VAE KAE VAE KAE VAE KAE VAE KAA VAA
KBBD VBBD KBA VBA KAE VAE KAE VAE KAA VAA
KBBD VBBD KBA VBA KAE VAE KAA VAA
KBBD VBBD KBA VBA KBBD VBBD KBA VBA KAE VAE KAA VAA
KBG VBG
KBBD KBA VBA KBBD VBBD KBA VBA KAE VAE KAE VAE KAA VAA
KBBD VBBD KBA VBA KBBD KBA VBA KBBD VBBD KBA VBA KBBD VBBD KBA
KAE VAE KAA VAA
KBBD VBBD KBA VBA KAE VAE KAE VAE VAE KAA VAA
📝 Markdown Zelle #17
Contrary to cognitivist models
(ARS, Koop, P. Grammar Induction, Parser, Grammar Transduction)
such a large language model explains nothing and therefore becomes
Large language model of postmodernism, posthumanism and transhumanism
celebrated with parasitic intent.
If you want to write a textbook on the rules of sales pitches,
but gets a software agent who likes to make sales calls,
you have done a bad job at a very high level.
(ARS, Koop, P. Grammar Induction, Parser, Grammar Transduction)
such a large language model explains nothing and therefore becomes
Large language model of postmodernism, posthumanism and transhumanism
celebrated with parasitic intent.
If you want to write a textbook on the rules of sales pitches,
but gets a software agent who likes to make sales calls,
you have done a bad job at a very high level.