Creating a generator function for a large data set for a neural networkSetting MaxIterations for neural network function `Predict`?Neural Network for polynomial fitImplementing a Neural NetworkHow to make similar objects have similar embedding weights?Neural network giving different answersImage Generator with a Neural Network (CIFAR-10)Backpropagation for manual neural network trainingMonitoring the Training Progress through PutAppendCreating an indicator function in a neural networkMatlab Neural Network Toolbox vs Wolfram Neural Network Framework
Where would I need my direct neural interface to be implanted?
What is the most common color to indicate the input-field is disabled?
Avoiding the "not like other girls" trope?
Label inside tikzcd square
Implication of namely
How seriously should I take size and weight limits of hand luggage?
How can I deal with my CEO asking me to hire someone with a higher salary than me, a co-founder?
Getting extremely large arrows with tikzcd
In the UK, is it possible to get a referendum by a court decision?
Why were 5.25" floppy drives cheaper than 8"?
How obscure is the use of 令 in 令和?
Were days ever written as ordinal numbers when writing day-month-year?
How to remove border from elements in the last row?
OP Amp not amplifying audio signal
Do creatures with a speed 0ft., fly 30ft. (hover) ever touch the ground?
What is the opposite of "eschatology"?
Did 'Cinema Songs' exist during Hiranyakshipu's time?
Is it "common practice in Fourier transform spectroscopy to multiply the measured interferogram by an apodizing function"? If so, why?
What is an equivalently powerful replacement spell for Yuan-Ti's Suggestion spell?
What is a Samsaran Word™?
Is it possible to map the firing of neurons in the human brain so as to stimulate artificial memories in someone else?
What is required to make GPS signals available indoors?
How do I exit BASH while loop using modulus operator?
Machine learning testing data
Creating a generator function for a large data set for a neural network
Setting MaxIterations for neural network function `Predict`?Neural Network for polynomial fitImplementing a Neural NetworkHow to make similar objects have similar embedding weights?Neural network giving different answersImage Generator with a Neural Network (CIFAR-10)Backpropagation for manual neural network trainingMonitoring the Training Progress through PutAppendCreating an indicator function in a neural networkMatlab Neural Network Toolbox vs Wolfram Neural Network Framework
$begingroup$
After reading documentation on Training on Large Data Sets, I am still at a loss on how to create a generator function, to read a batch of 1000 of this
1, 17, 39, 53, 44, 23 -> 18, 53, 50, 38, 6, 31
For an entire 66GB file. I have tried
ReadList[$fileStream, Expression,1000]
but this only reads the first 1000.
Any assistance would be appreciated.
Also, any good books on this subject?
neural-networks
edited Mar 28 at 14:17
m_goldberg
88.1k872199
asked Mar 28 at 12:39
Michel MesedahlMichel Mesedahl
262
$endgroup$
add a comment |
$begingroup$
After reading documentation on Training on Large Data Sets, I am still at a loss on how to create a generator function, to read a batch of 1000 of this
1, 17, 39, 53, 44, 23 -> 18, 53, 50, 38, 6, 31
For an entire 66GB file. I have tried
ReadList[$fileStream, Expression,1000]
but this only reads the first 1000.
Any assistance would be appreciated.
Also, any good books on this subject?
neural-networks
edited Mar 28 at 14:17
m_goldberg
88.1k872199
asked Mar 28 at 12:39
Michel MesedahlMichel Mesedahl
262
$endgroup$
add a comment |
$begingroup$
After reading documentation on Training on Large Data Sets, I am still at a loss on how to create a generator function, to read a batch of 1000 of this
1, 17, 39, 53, 44, 23 -> 18, 53, 50, 38, 6, 31
For an entire 66GB file. I have tried
ReadList[$fileStream, Expression,1000]
but this only reads the first 1000.
Any assistance would be appreciated.
Also, any good books on this subject?
neural-networks
edited Mar 28 at 14:17
m_goldberg
88.1k872199
asked Mar 28 at 12:39
Michel MesedahlMichel Mesedahl
262
$endgroup$
After reading documentation on Training on Large Data Sets, I am still at a loss on how to create a generator function, to read a batch of 1000 of this
1, 17, 39, 53, 44, 23 -> 18, 53, 50, 38, 6, 31
For an entire 66GB file. I have tried
ReadList[$fileStream, Expression,1000]
but this only reads the first 1000.
Any assistance would be appreciated.
Also, any good books on this subject?
neural-networks
neural-networks
edited Mar 28 at 14:17
m_goldberg
88.1k872199
asked Mar 28 at 12:39
Michel MesedahlMichel Mesedahl
262
edited Mar 28 at 14:17
m_goldberg
88.1k872199
asked Mar 28 at 12:39
Michel MesedahlMichel Mesedahl
262
edited Mar 28 at 14:17
m_goldberg
88.1k872199
edited Mar 28 at 14:17
m_goldberg
88.1k872199
edited Mar 28 at 14:17
m_goldberg
88.1k872199
88.1k872199
asked Mar 28 at 12:39
Michel MesedahlMichel Mesedahl
262
asked Mar 28 at 12:39
Michel MesedahlMichel Mesedahl
262
asked Mar 28 at 12:39
Michel MesedahlMichel Mesedahl
262
262
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
$begingroup$
f = OpenWrite["data.txt"];
SeedRandom[0];
Do[Write[f, RandomInteger[0, 100, 5] -> RandomInteger[0, 100, 5]], 10000]
Close[f];
f = OpenRead["data.txt"];
generator = Function[
Table[
With[
r = Read[f, Record],
If[r === EndOfFile,
SetStreamPosition[f, 0]; Read[f, Record],
r
]
],
#BatchSize
] // ToExpression // <|"Input" -> #[[;; , 1]], "Output" -> #[[;; , 2]]|> &
];
net = NetChain[
LinearLayer[16],
LinearLayer[5]
,
"Input" -> 5,
"Output" -> 5
]
SetStreamPosition[f, 0];
netT = NetTrain[
net,
generator, "RoundLength" -> 10,
All,
BatchSize -> 1000, MaxTrainingRounds -> 10
]
answered Mar 28 at 13:43
Alexey GolyshevAlexey Golyshev
5,1071739
$endgroup$
$begingroup$
Note: Mathematica seems to ignoreRoundLengthfor my generator. Play withMaxTrainingRounds.
$endgroup$
– Alexey Golyshev
Mar 28 at 14:19
$begingroup$
Hello, I tried your above code. This also only runs through 1000 entries , the batch does not progress through the whole file. Any suggestions?
$endgroup$
– Michel Mesedahl
Mar 30 at 12:52
$begingroup$
@MichelMesedahl How did you measure? My generator works cyclically. 1drv.ms/u/s!AvVas0AkeAWi0RHJDhoL5DFtsl6l (Print@StreamPosition[f]inserted in generator). For your 66 GB file, you must changeMaxTrainingRounds(to millions or billions).
$endgroup$
– Alexey Golyshev
Mar 30 at 17:43
$begingroup$
Ok, now I get the jist of it. Thank You
$endgroup$
– Michel Mesedahl
Mar 30 at 18:01
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
return StackExchange.using("mathjaxEditing", function ()
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix)
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
);
);
, "mathjax-editing");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "387"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmathematica.stackexchange.com%2fquestions%2f194098%2fcreating-a-generator-function-for-a-large-data-set-for-a-neural-network%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
$begingroup$
f = OpenWrite["data.txt"];
SeedRandom[0];
Do[Write[f, RandomInteger[0, 100, 5] -> RandomInteger[0, 100, 5]], 10000]
Close[f];
f = OpenRead["data.txt"];
generator = Function[
Table[
With[
r = Read[f, Record],
If[r === EndOfFile,
SetStreamPosition[f, 0]; Read[f, Record],
r
]
],
#BatchSize
] // ToExpression // <|"Input" -> #[[;; , 1]], "Output" -> #[[;; , 2]]|> &
];
net = NetChain[
LinearLayer[16],
LinearLayer[5]
,
"Input" -> 5,
"Output" -> 5
]
SetStreamPosition[f, 0];
netT = NetTrain[
net,
generator, "RoundLength" -> 10,
All,
BatchSize -> 1000, MaxTrainingRounds -> 10
]
answered Mar 28 at 13:43
Alexey GolyshevAlexey Golyshev
5,1071739
$endgroup$
$begingroup$
Note: Mathematica seems to ignoreRoundLengthfor my generator. Play withMaxTrainingRounds.
$endgroup$
– Alexey Golyshev
Mar 28 at 14:19
$begingroup$
Hello, I tried your above code. This also only runs through 1000 entries , the batch does not progress through the whole file. Any suggestions?
$endgroup$
– Michel Mesedahl
Mar 30 at 12:52
$begingroup$
@MichelMesedahl How did you measure? My generator works cyclically. 1drv.ms/u/s!AvVas0AkeAWi0RHJDhoL5DFtsl6l (Print@StreamPosition[f]inserted in generator). For your 66 GB file, you must changeMaxTrainingRounds(to millions or billions).
$endgroup$
– Alexey Golyshev
Mar 30 at 17:43
$begingroup$
Ok, now I get the jist of it. Thank You
$endgroup$
– Michel Mesedahl
Mar 30 at 18:01
add a comment |
$begingroup$
f = OpenWrite["data.txt"];
SeedRandom[0];
Do[Write[f, RandomInteger[0, 100, 5] -> RandomInteger[0, 100, 5]], 10000]
Close[f];
f = OpenRead["data.txt"];
generator = Function[
Table[
With[
r = Read[f, Record],
If[r === EndOfFile,
SetStreamPosition[f, 0]; Read[f, Record],
r
]
],
#BatchSize
] // ToExpression // <|"Input" -> #[[;; , 1]], "Output" -> #[[;; , 2]]|> &
];
net = NetChain[
LinearLayer[16],
LinearLayer[5]
,
"Input" -> 5,
"Output" -> 5
]
SetStreamPosition[f, 0];
netT = NetTrain[
net,
generator, "RoundLength" -> 10,
All,
BatchSize -> 1000, MaxTrainingRounds -> 10
]
answered Mar 28 at 13:43
Alexey GolyshevAlexey Golyshev
5,1071739
$endgroup$
$begingroup$
Note: Mathematica seems to ignoreRoundLengthfor my generator. Play withMaxTrainingRounds.
$endgroup$
– Alexey Golyshev
Mar 28 at 14:19
$begingroup$
Hello, I tried your above code. This also only runs through 1000 entries , the batch does not progress through the whole file. Any suggestions?
$endgroup$
– Michel Mesedahl
Mar 30 at 12:52
$begingroup$
@MichelMesedahl How did you measure? My generator works cyclically. 1drv.ms/u/s!AvVas0AkeAWi0RHJDhoL5DFtsl6l (Print@StreamPosition[f]inserted in generator). For your 66 GB file, you must changeMaxTrainingRounds(to millions or billions).
$endgroup$
– Alexey Golyshev
Mar 30 at 17:43
$begingroup$
Ok, now I get the jist of it. Thank You
$endgroup$
– Michel Mesedahl
Mar 30 at 18:01
add a comment |
$begingroup$
f = OpenWrite["data.txt"];
SeedRandom[0];
Do[Write[f, RandomInteger[0, 100, 5] -> RandomInteger[0, 100, 5]], 10000]
Close[f];
f = OpenRead["data.txt"];
generator = Function[
Table[
With[
r = Read[f, Record],
If[r === EndOfFile,
SetStreamPosition[f, 0]; Read[f, Record],
r
]
],
#BatchSize
] // ToExpression // <|"Input" -> #[[;; , 1]], "Output" -> #[[;; , 2]]|> &
];
net = NetChain[
LinearLayer[16],
LinearLayer[5]
,
"Input" -> 5,
"Output" -> 5
]
SetStreamPosition[f, 0];
netT = NetTrain[
net,
generator, "RoundLength" -> 10,
All,
BatchSize -> 1000, MaxTrainingRounds -> 10
]
answered Mar 28 at 13:43
Alexey GolyshevAlexey Golyshev
5,1071739
$endgroup$
f = OpenWrite["data.txt"];
SeedRandom[0];
Do[Write[f, RandomInteger[0, 100, 5] -> RandomInteger[0, 100, 5]], 10000]
Close[f];
f = OpenRead["data.txt"];
generator = Function[
Table[
With[
r = Read[f, Record],
If[r === EndOfFile,
SetStreamPosition[f, 0]; Read[f, Record],
r
]
],
#BatchSize
] // ToExpression // <|"Input" -> #[[;; , 1]], "Output" -> #[[;; , 2]]|> &
];
net = NetChain[
LinearLayer[16],
LinearLayer[5]
,
"Input" -> 5,
"Output" -> 5
]
SetStreamPosition[f, 0];
netT = NetTrain[
net,
generator, "RoundLength" -> 10,
All,
BatchSize -> 1000, MaxTrainingRounds -> 10
]
answered Mar 28 at 13:43
Alexey GolyshevAlexey Golyshev
5,1071739
answered Mar 28 at 13:43
Alexey GolyshevAlexey Golyshev
5,1071739
answered Mar 28 at 13:43
Alexey GolyshevAlexey Golyshev
5,1071739
answered Mar 28 at 13:43
Alexey GolyshevAlexey Golyshev
5,1071739
5,1071739
$begingroup$
Note: Mathematica seems to ignoreRoundLengthfor my generator. Play withMaxTrainingRounds.
$endgroup$
– Alexey Golyshev
Mar 28 at 14:19
$begingroup$
Hello, I tried your above code. This also only runs through 1000 entries , the batch does not progress through the whole file. Any suggestions?
$endgroup$
– Michel Mesedahl
Mar 30 at 12:52
$begingroup$
@MichelMesedahl How did you measure? My generator works cyclically. 1drv.ms/u/s!AvVas0AkeAWi0RHJDhoL5DFtsl6l (Print@StreamPosition[f]inserted in generator). For your 66 GB file, you must changeMaxTrainingRounds(to millions or billions).
$endgroup$
– Alexey Golyshev
Mar 30 at 17:43
$begingroup$
Ok, now I get the jist of it. Thank You
$endgroup$
– Michel Mesedahl
Mar 30 at 18:01
add a comment |
$begingroup$
Note: Mathematica seems to ignoreRoundLengthfor my generator. Play withMaxTrainingRounds.
$endgroup$
– Alexey Golyshev
Mar 28 at 14:19
$begingroup$
Hello, I tried your above code. This also only runs through 1000 entries , the batch does not progress through the whole file. Any suggestions?
$endgroup$
– Michel Mesedahl
Mar 30 at 12:52
$begingroup$
@MichelMesedahl How did you measure? My generator works cyclically. 1drv.ms/u/s!AvVas0AkeAWi0RHJDhoL5DFtsl6l (Print@StreamPosition[f]inserted in generator). For your 66 GB file, you must changeMaxTrainingRounds(to millions or billions).
$endgroup$
– Alexey Golyshev
Mar 30 at 17:43
$begingroup$
Ok, now I get the jist of it. Thank You
$endgroup$
– Michel Mesedahl
Mar 30 at 18:01
$begingroup$
Note: Mathematica seems to ignore
RoundLength for my generator. Play with MaxTrainingRounds.$endgroup$
– Alexey Golyshev
Mar 28 at 14:19
$begingroup$
Note: Mathematica seems to ignore
RoundLength for my generator. Play with MaxTrainingRounds.$endgroup$
– Alexey Golyshev
Mar 28 at 14:19
$begingroup$
Hello, I tried your above code. This also only runs through 1000 entries , the batch does not progress through the whole file. Any suggestions?
$endgroup$
– Michel Mesedahl
Mar 30 at 12:52
$begingroup$
Hello, I tried your above code. This also only runs through 1000 entries , the batch does not progress through the whole file. Any suggestions?
$endgroup$
– Michel Mesedahl
Mar 30 at 12:52
$begingroup$
@MichelMesedahl How did you measure? My generator works cyclically. 1drv.ms/u/s!AvVas0AkeAWi0RHJDhoL5DFtsl6l (
Print@StreamPosition[f] inserted in generator). For your 66 GB file, you must change MaxTrainingRounds (to millions or billions).$endgroup$
– Alexey Golyshev
Mar 30 at 17:43
$begingroup$
@MichelMesedahl How did you measure? My generator works cyclically. 1drv.ms/u/s!AvVas0AkeAWi0RHJDhoL5DFtsl6l (
Print@StreamPosition[f] inserted in generator). For your 66 GB file, you must change MaxTrainingRounds (to millions or billions).$endgroup$
– Alexey Golyshev
Mar 30 at 17:43
$begingroup$
Ok, now I get the jist of it. Thank You
$endgroup$
– Michel Mesedahl
Mar 30 at 18:01
$begingroup$
Ok, now I get the jist of it. Thank You
$endgroup$
– Michel Mesedahl
Mar 30 at 18:01
add a comment |
Thanks for contributing an answer to Mathematica Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmathematica.stackexchange.com%2fquestions%2f194098%2fcreating-a-generator-function-for-a-large-data-set-for-a-neural-network%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
